03/24/2025 00:47:37.568 7f56f65a6d08 [contrail-device-manager] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/24/2025 00:47:37.568 7f56f65a6d08 [contrail-device-manager] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/24/2025 00:47:37.568 7f56f65a6d08 [contrail-device-manager] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/24/2025 00:47:37.568 7f56edc84390 [contrail-device-manager] [INFO]: Processing event[EvStart] in state[none] 03/24/2025 00:47:37.568 7f56edc84390 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:47:37.569 7f56edc84390 [contrail-device-manager] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/24/2025 00:47:37.569 7f56edc84390 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/24/2025 00:47:37.569 7f56edc84390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.569 7f56edc84390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/24/2025 00:47:37.569 7f56edc84390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.569 7f56edc84390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.569 7f56edc84390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1742777257569153 collector_name = collector_ip = collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 1 >> 03/24/2025 00:47:37.569 7f56edc84390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.618 7f56f65a6d08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//', 'pyamqp://guest:guest@10.0.0.57:5673//'] 03/24/2025 00:47:37.627 7f56f65a6d08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 03/24/2025 00:47:37.628 7f56edbd6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.628 7f56edbd6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/24/2025 00:47:37.628 7f56edbd6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.628 7f56edbd6630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.628 7f56edbd6630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/24/2025 00:47:37.628 7f56edbd6630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.633 7f56f65a6d08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 03/24/2025 00:47:37.633 7f56edbd62e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.633 7f56edbd62e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/24/2025 00:47:37.633 7f56edbd62e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.690 7f56f65a6d08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 03/24/2025 00:47:37.691 7f56edbd68d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.691 7f56edbd68d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 03/24/2025 00:47:37.691 7f56edbd68d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.698 7f56f65a6d08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 03/24/2025 00:47:37.698 7f56fc489978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.698 7f56fc489978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/24/2025 00:47:37.698 7f56fc489978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.732 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Removed remaining AMQP queue from previous run 03/24/2025 00:47:37.732 7f56f65a6d08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 03/24/2025 00:47:37.732 7f56f65a6d08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 03/24/2025 00:47:37.733 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 03/24/2025 00:47:37.733 7f56f350f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting consumer greenlet 03/24/2025 00:47:37.733 7f56f350f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 03/24/2025 00:47:37.733 7f56f041da20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Starting publisher greenlet 03/24/2025 00:47:37.733 7f56f041da20 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Publisher connection 03/24/2025 00:47:37.734 7f56eda70240 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.734 7f56eda70240 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:47:37.734 7f56eda70240 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.736 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 03/24/2025 00:47:37.742 7f56f041da20 [contrail-device-manager] [INFO]: KombuAmqpClient: Publisher connection established 03/24/2025 00:47:37.742 7f56f350f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 03/24/2025 00:47:37.743 7f56f350f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Waiting for consumer 03/24/2025 00:47:37.779 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 03/24/2025 00:47:37.782 7f56ed9c2ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.782 7f56ed9c2ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 03/24/2025 00:47:37.782 7f56ed9c2ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.795 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 03/24/2025 00:47:37.795 7f56edaa0ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:37.796 7f56edaa0ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:47:37.796 7f56edaa0ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:38.802 7f56f65a6d08 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 03/24/2025 00:47:39.813 7f56f65a6d08 [contrail-device-manager] [WARNING]: Waiting for keyspace config_db_uuid to be created 03/24/2025 00:47:39.819 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 03/24/2025 00:47:39.844 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 03/24/2025 00:47:40.634 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:47:41.569 7f56ed952048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:47:41.571 7f56ed952048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:47:41.571 7f56ed952048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:47:41.571 7f56ed952048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:47:41.571 7f56ed952048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:47:41.571 7f56ed952048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:47:41.571 7f56ed952048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:47:41.572 7f56ed952048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:47:41.572 7f56ed952048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:41.572 7f56ed952048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:47:41.572 7f56ed952048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:41.572 7f56ed952048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:41.572 7f56ed952048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:47:41.572 7f56ed952048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:41.578 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'obj_uuid_table', created 03/24/2025 00:47:41.585 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:47:42.846 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:47:43.708 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 03/24/2025 00:47:43.714 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:47:45.571 7f56ed0eb6d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:47:45.573 7f56ed0eb6d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:47:45.573 7f56ed0eb6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:47:45.573 7f56ed0eb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:47:45.574 7f56ed0eb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:47:45.574 7f56ed0eb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:47:45.574 7f56ed0eb6d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:47:45.574 7f56ed0eb6d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:47:45.574 7f56ed0eb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:45.574 7f56ed0eb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:47:45.574 7f56ed0eb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:45.574 7f56ed0eb6d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:45.574 7f56ed0eb6d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:47:45.574 7f56ed0eb6d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:46.430 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:47:47.412 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table', created 03/24/2025 00:47:47.422 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:47:48.776 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:47:49.574 7f56ed948630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:47:49.575 7f56ed948630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:47:49.575 7f56ed948630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:47:49.575 7f56ed948630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:47:49.575 7f56ed948630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:47:49.575 7f56ed948630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:47:49.575 7f56ed948630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:47:49.576 7f56ed948630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:47:49.576 7f56ed948630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:49.576 7f56ed948630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:47:49.576 7f56ed948630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:49.576 7f56ed948630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:49.576 7f56ed948630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:47:49.576 7f56ed948630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:49.721 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 03/24/2025 00:47:49.727 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:47:51.351 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:47:51.353 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'obj_shared_table', created 03/24/2025 00:47:51.360 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:47:52.870 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:47:53.576 7f56ed952630 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:47:53.577 7f56ed952630 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:47:53.577 7f56ed952630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:47:53.577 7f56ed952630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:47:53.577 7f56ed952630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:47:53.578 7f56ed952630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:47:53.578 7f56ed952630 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:47:53.578 7f56ed952630 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:47:53.578 7f56ed952630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:53.578 7f56ed952630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:47:53.578 7f56ed952630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:53.578 7f56ed952630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:53.578 7f56ed952630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:47:53.578 7f56ed952630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:53.778 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 03/24/2025 00:47:53.784 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:47:53.784 7f56ed963e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:53.784 7f56ed963e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:47:53.785 7f56ed963e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:53.807 7f56f65a6d08 [contrail-device-manager] [ERROR]: Error in db walk read 'type' 03/24/2025 00:47:53.808 7f56ed9524e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:53.809 7f56ed9524e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Zookeeper:Zookeeper[] connection down >>, ] >> 03/24/2025 00:47:53.809 7f56ed9524e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:53.819 7f56ed0f8eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:53.819 7f56ed0f8eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:47:53.820 7f56ed0f8eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:53.820 7f56f65a6d08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_status_consumer.dummy 03/24/2025 00:47:53.820 7f56f65a6d08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_request_consumer 03/24/2025 00:47:53.820 7f56f350f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 03/24/2025 00:47:53.821 7f56f65a6d08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2938-1. 03/24/2025 00:47:53.829 7f56f350f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 03/24/2025 00:47:54.016 7f56f350f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2938-1.'] 03/24/2025 00:47:54.041 7f56f350f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 03/24/2025 00:47:54.045 7f56f350f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 03/24/2025 00:47:54.133 7f56f350f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2938-1.'] 03/24/2025 00:47:54.341 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Fetching all containers 03/24/2025 00:47:54.517 7f56f65a6d08 [contrail-device-manager] [INFO]: Restarting dnsmasq docker: config_dnsmasq_1 03/24/2025 00:47:54.747 7f56f65a6d08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2938-1..config_queue 03/24/2025 00:47:54.748 7f56f65a6d08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2938-1..tftp_queue 03/24/2025 00:47:55.159 7f56f350f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 03/24/2025 00:47:55.166 7f56f350f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 03/24/2025 00:47:55.248 7f56f65a6d08 [contrail-device-manager] [WARNING]: Process 136 prepared to run in Full mode... 03/24/2025 00:47:55.248 7f56f65a6d08 [contrail-device-manager] [WARNING]: Elected master Device Manager node. Initializing... 03/24/2025 00:47:55.248 7f56f65a6d08 [contrail-device-manager] [INFO]: SANDESH: INTROSPECT IS ON: 0.0.0.0:8096 03/24/2025 00:47:55.251 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add [config_updater.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/config_updater] to home page 03/24/2025 00:47:55.251 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_ConfigApiUpdateReq] 03/24/2025 00:47:55.252 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add [greenlets.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/greenlets] to home page 03/24/2025 00:47:55.252 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_GreenletObjectReq] 03/24/2025 00:47:55.253 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add [cpuinfo.xml:/usr/lib/python3.6/site-packages/cfgm_common/uve/nodeinfo/cpuinfo] to home page 03/24/2025 00:47:55.253 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CpuLoadInfoReq] 03/24/2025 00:47:55.257 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add [dm_introspect.xml:/usr/lib/python3.6/site-packages/device_manager/sandesh/dm_introspect] to home page 03/24/2025 00:47:55.257 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_RoutingInstanceList] 03/24/2025 00:47:55.257 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualNetworkList] 03/24/2025 00:47:55.257 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_VirtualMachineInterfaceList] 03/24/2025 00:47:55.257 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalInterfaceList] 03/24/2025 00:47:55.257 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalInterfaceList] 03/24/2025 00:47:55.257 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PhysicalRouterList] 03/24/2025 00:47:55.257 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_BgpRouterList] 03/24/2025 00:47:55.257 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_LogicalRouterList] 03/24/2025 00:47:55.257 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_InstanceIpList] 03/24/2025 00:47:55.257 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FloatingIpList] 03/24/2025 00:47:55.264 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_PortTupleList] 03/24/2025 00:47:55.264 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_FeatureFlagList] 03/24/2025 00:47:55.266 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add [sandesh_alarm.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_alarm] to home page 03/24/2025 00:47:55.266 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmCacheRequest] 03/24/2025 00:47:55.266 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmTypesRequest] 03/24/2025 00:47:55.266 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshAlarmAckRequest] 03/24/2025 00:47:55.266 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add [sandesh_ctrl.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_ctrl] to home page 03/24/2025 00:47:55.266 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlClientToServer] 03/24/2025 00:47:55.266 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshCtrlServerToClient] 03/24/2025 00:47:55.267 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add [sandesh_trace.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_trace] to home page 03/24/2025 00:47:55.267 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceRequest] 03/24/2025 00:47:55.267 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferListRequest] 03/24/2025 00:47:55.267 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceEnableDisableReq] 03/24/2025 00:47:55.267 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufStatusReq] 03/24/2025 00:47:55.267 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshTraceBufferEnableDisableReq] 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add [sandesh_uve.xml:/usr/lib/python3.6/site-packages/pysandesh/gen_py/sandesh_uve] to home page 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshMessageStatsReq] 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsSet] 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshLoggingParamsStatus] 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsSet] 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendingParamsStatus] 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsSet] 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsReset] 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueParamsStatus] 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueSet] 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshSendQueueStatus] 03/24/2025 00:47:55.268 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_CollectorInfoRequest] 03/24/2025 00:47:55.269 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVEDSConfReq] 03/24/2025 00:47:55.271 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVECacheReq] 03/24/2025 00:47:55.271 7f56f65a6d08 [contrail-device-manager] [DEBUG]: Add http request [/Snh_SandeshUVETypesReq] 03/24/2025 00:47:55.273 7f56f65a6d08 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Added consumer: device_manager_ztp.ztp_queue 03/24/2025 00:47:55.388 7f56ed0d7240 [contrail-device-manager] [ERROR]: Cannot write http_port 8096 to /tmp/contrail-device-manager.1.http_port 03/24/2025 00:47:55.388 7f56ed0d7240 [contrail-device-manager] [ERROR]: Starting Introspect on HTTP Port 8096 03/24/2025 00:47:55.388 7f56ecf29c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:55.395 7f56ecf29c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Initializing description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 03/24/2025 00:47:55.395 7f56ecf29c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:55.416 7f56ec76aeb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:55.417 7f56ec76aeb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 03/24/2025 00:47:55.417 7f56ec76aeb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:55.702 7f56f350f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2938-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2938-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2938-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 03/24/2025 00:47:55.737 7f56f350f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Ensuring Consumer connection 03/24/2025 00:47:55.745 7f56f350f780 [contrail-device-manager] [INFO]: KombuAmqpClient: Consumer connection established 03/24/2025 00:47:55.809 7f56f350f780 [contrail-device-manager] [DEBUG]: KombuAmqpClient: Created consumers ['job_status_consumer.dummy', 'job_request_consumer', 'job_abort_consumer.cn-jenkins-deploy-platform-ansible-os-2938-1.', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2938-1..config_queue', 'device_manager_ztp.cn-jenkins-deploy-platform-ansible-os-2938-1..tftp_queue', 'device_manager_ztp.ztp_queue'] 03/24/2025 00:47:57.578 7f56ec7714e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:47:57.580 7f56ec7714e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:47:57.580 7f56ec7714e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:47:57.580 7f56ec7714e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:47:57.580 7f56ec7714e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 03/24/2025 00:47:57.580 7f56ec7714e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:47:57.580 7f56ec7714e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:47:57.581 7f56ec7714e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:47:57.581 7f56ec7714e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:57.581 7f56ec7714e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Down description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], ApiServer:ApiServer[ApiServer Connection State updated] connection down >>, ] >> 03/24/2025 00:47:57.581 7f56ec7714e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:57.581 7f56ec7714e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:47:57.581 7f56ec7714e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:47:57.581 7f56ec7714e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:01.582 7f56ec7718d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:01.583 7f56ec7718d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:01.583 7f56ec7718d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:01.583 7f56ec7718d0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:01.586 7f56ec7718d0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:01.586 7f56ec7718d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:01.586 7f56ec7718d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:01.586 7f56ec7718d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:01.586 7f56ec7718d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:01.594 7f56ec7718d0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:01.595 7f56ec7718d0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:01.595 7f56ec7718d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:01.595 7f56ec7718d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:01.595 7f56ec7718d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:01.595 7f56ec7718d0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:01.595 7f56ec7718d0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:01.595 7f56ec7718d0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:05.587 7f56ec619e10 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:05.589 7f56ec619e10 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:05.589 7f56ec619e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:05.589 7f56ec619e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:05.590 7f56ec619e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:05.590 7f56ec619e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:05.590 7f56ec619e10 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:05.590 7f56ec619e10 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:05.590 7f56ec619e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:05.590 7f56ec619e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:05.590 7f56ec619e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:05.590 7f56ec619e10 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:05.590 7f56ec619e10 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:05.590 7f56ec619e10 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:09.590 7f56ec615cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:09.592 7f56ec615cc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:09.592 7f56ec615cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:09.592 7f56ec615cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:09.592 7f56ec615cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:09.592 7f56ec615cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:09.592 7f56ec615cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:09.593 7f56ec615cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:09.593 7f56ec615cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:09.593 7f56ec615cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:09.593 7f56ec615cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:09.593 7f56ec615cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:09.593 7f56ec615cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:09.593 7f56ec615cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:13.598 7f56edbd60f0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:13.605 7f56edbd60f0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:13.605 7f56edbd60f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:13.605 7f56edbd60f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:13.606 7f56edbd60f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:13.606 7f56edbd60f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:13.606 7f56edbd60f0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:13.606 7f56edbd60f0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:13.606 7f56edbd60f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:13.606 7f56edbd60f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:13.606 7f56edbd60f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:13.606 7f56edbd60f0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:13.606 7f56edbd60f0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:13.607 7f56edbd60f0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:17.606 7f56ed97fa20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:17.607 7f56ed97fa20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:17.607 7f56ed97fa20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:17.607 7f56ed97fa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:17.607 7f56ed97fa20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:17.607 7f56ed97fa20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:17.607 7f56ed97fa20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:17.607 7f56ed97fa20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:17.607 7f56ed97fa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:17.607 7f56ed97fa20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:17.607 7f56ed97fa20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:17.608 7f56ed97fa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:17.608 7f56ed97fa20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:17.608 7f56ed97fa20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:21.607 7f56ec775ac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:21.608 7f56ec775ac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:21.608 7f56ec775ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:21.608 7f56ec775ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:21.608 7f56ec775ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:21.608 7f56ec775ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:21.609 7f56ec775ac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:21.609 7f56ec775ac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:21.609 7f56ec775ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:21.609 7f56ec775ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:21.609 7f56ec775ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:21.609 7f56ec775ac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:21.609 7f56ec775ac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:21.609 7f56ec775ac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:25.609 7f56ec620b70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:25.610 7f56ec620b70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:25.610 7f56ec620b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:25.611 7f56ec620b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:25.611 7f56ec620b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:25.611 7f56ec620b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:25.611 7f56ec620b70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:25.611 7f56ec620b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:25.611 7f56ec620b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:25.611 7f56ec620b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:25.611 7f56ec620b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:25.611 7f56ec620b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:25.611 7f56ec620b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:25.612 7f56ec620b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:29.614 7f56ec624828 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:29.618 7f56ec624828 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:29.618 7f56ec624828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:29.618 7f56ec624828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:29.618 7f56ec624828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:29.618 7f56ec624828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:29.619 7f56ec624828 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:29.619 7f56ec624828 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:29.619 7f56ec624828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:29.619 7f56ec624828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:29.619 7f56ec624828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:29.619 7f56ec624828 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:29.619 7f56ec624828 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:29.619 7f56ec624828 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:33.619 7f56ec617a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:33.620 7f56ec617a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:33.620 7f56ec617a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:33.620 7f56ec617a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:33.621 7f56ec617a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:33.621 7f56ec617a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:33.621 7f56ec617a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:33.621 7f56ec617a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:33.621 7f56ec617a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:33.621 7f56ec617a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:33.621 7f56ec617a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:33.621 7f56ec617a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:33.621 7f56ec617a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:33.621 7f56ec617a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:37.623 7f56ec620c18 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:37.626 7f56ec620c18 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:37.627 7f56ec620c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:37.627 7f56ec620c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:37.627 7f56ec620c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:37.627 7f56ec620c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:37.627 7f56ec620c18 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:37.627 7f56ec620c18 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:37.627 7f56ec620c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:37.628 7f56ec620c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:37.628 7f56ec620c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:37.628 7f56ec620c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:37.628 7f56ec620c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:37.628 7f56ec620c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:41.627 7f56ec76aac8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:41.628 7f56ec76aac8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:41.628 7f56ec76aac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:41.628 7f56ec76aac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:41.629 7f56ec76aac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:41.629 7f56ec76aac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:41.629 7f56ec76aac8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:41.629 7f56ec76aac8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:41.629 7f56ec76aac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:41.629 7f56ec76aac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:41.629 7f56ec76aac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:41.629 7f56ec76aac8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:41.629 7f56ec76aac8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:41.629 7f56ec76aac8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:45.629 7f56ec7754e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:45.630 7f56ec7754e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:45.631 7f56ec7754e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:45.631 7f56ec7754e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:45.631 7f56ec7754e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:45.631 7f56ec7754e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:45.631 7f56ec7754e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:45.631 7f56ec7754e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:45.631 7f56ec7754e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:45.631 7f56ec7754e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:45.631 7f56ec7754e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:45.632 7f56ec7754e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:45.632 7f56ec7754e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:45.632 7f56ec7754e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:49.631 7f56ed0fed68 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:49.633 7f56ed0fed68 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:49.633 7f56ed0fed68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:49.633 7f56ed0fed68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:49.634 7f56ed0fed68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:49.634 7f56ed0fed68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:49.634 7f56ed0fed68 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:49.635 7f56ed0fed68 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:49.635 7f56ed0fed68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:49.635 7f56ed0fed68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:49.635 7f56ed0fed68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:49.635 7f56ed0fed68 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:49.635 7f56ed0fed68 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:49.635 7f56ed0fed68 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:53.634 7f56ec76aa20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:53.635 7f56ec76aa20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:53.635 7f56ec76aa20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:53.635 7f56ec76aa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:53.635 7f56ec76aa20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:53.635 7f56ec76aa20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:53.635 7f56ec76aa20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:53.635 7f56ec76aa20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:53.636 7f56ec76aa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:53.636 7f56ec76aa20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:53.636 7f56ec76aa20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:53.636 7f56ec76aa20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:53.636 7f56ec76aa20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:53.636 7f56ec76aa20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:57.636 7f56ec620b70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:48:57.637 7f56ec620b70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:48:57.637 7f56ec620b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:48:57.637 7f56ec620b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:57.637 7f56ec620b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:48:57.637 7f56ec620b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:48:57.637 7f56ec620b70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:48:57.638 7f56ec620b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:48:57.638 7f56ec620b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:57.638 7f56ec620b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:48:57.638 7f56ec620b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:57.638 7f56ec620b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:57.638 7f56ec620b70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:48:57.638 7f56ec620b70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:59.371 7f56f65a6d08 [contrail-device-manager] [WARNING]: VNCCassandra started with driver cql 03/24/2025 00:48:59.371 7f56f65a6d08 [contrail-device-manager] [WARNING]: initializing cassandra cluster 03/24/2025 00:48:59.371 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster before create 03/24/2025 00:48:59.371 7f56ec765438 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:59.372 7f56ec765438 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:48:59.372 7f56ec765438 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:59.373 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster created 03/24/2025 00:48:59.418 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster DB connected 03/24/2025 00:48:59.420 7f56ec5d7c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:59.420 7f56ec5d7c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Initializing description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[], Zookeeper:Zookeeper[] connection down >>, ] >> 03/24/2025 00:48:59.420 7f56ec5d7c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:59.431 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster init KS 03/24/2025 00:48:59.431 7f56ec5d7a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:59.432 7f56ec5d7a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:48:59.432 7f56ec5d7a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:48:59.453 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:48:59.741 7f56f65a6d08 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace', created 03/24/2025 00:48:59.747 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:48:59.756 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:49:01.438 7f56f65a6d08 [contrail-device-manager] [WARNING]: keyspace 'dm_keyspace' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 03/24/2025 00:49:01.443 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:49:01.443 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster KS are ready 03/24/2025 00:49:01.552 7f56f65a6d08 [contrail-device-manager] [WARNING]: CassandraDriverCQL.__init__Cluster CF are ready 03/24/2025 00:49:01.560 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:49:01.638 7f56ec7284e0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:01.639 7f56ec7284e0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:01.639 7f56ec7284e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:01.639 7f56ec7284e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:01.639 7f56ec7284e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:49:01.641 7f56ec7284e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:01.641 7f56ec7284e0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:01.641 7f56ec7284e0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:01.641 7f56ec7284e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:01.641 7f56ec7284e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:49:01.642 7f56ec7284e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:01.642 7f56ec7284e0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:01.642 7f56ec7284e0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:01.642 7f56ec7284e0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:03.368 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table', created 03/24/2025 00:49:03.373 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:49:03.382 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:49:04.506 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'dm_pr_vn_ip_table' fixed with properties {'gc_grace_seconds': 864000} 03/24/2025 00:49:04.511 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:49:04.518 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:49:05.640 7f56ec53fb70 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:05.641 7f56ec53fb70 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:05.641 7f56ec53fb70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:05.641 7f56ec53fb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:05.641 7f56ec53fb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:49:05.641 7f56ec53fb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:05.641 7f56ec53fb70 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:05.642 7f56ec53fb70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:05.642 7f56ec53fb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:05.642 7f56ec53fb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:49:05.642 7f56ec53fb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:05.642 7f56ec53fb70 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:05.642 7f56ec53fb70 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:05.642 7f56ec53fb70 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:05.767 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table', created 03/24/2025 00:49:05.772 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:49:05.779 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:49:06.695 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'dm_pr_asn_table' fixed with properties {'gc_grace_seconds': 864000} 03/24/2025 00:49:06.701 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:49:06.710 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:49:08.483 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table', created 03/24/2025 00:49:08.489 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:49:08.496 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:49:08.790 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'dm_ni_ipv6_ll_table' fixed with properties {'gc_grace_seconds': 864000} 03/24/2025 00:49:08.798 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:49:08.808 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:49:09.641 7f56ec55c2e8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:09.643 7f56ec55c2e8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:09.643 7f56ec55c2e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:09.643 7f56ec55c2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:09.643 7f56ec55c2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:49:09.643 7f56ec55c2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:09.643 7f56ec55c2e8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:09.643 7f56ec55c2e8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:09.643 7f56ec55c2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:09.644 7f56ec55c2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Initializing description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 03/24/2025 00:49:09.644 7f56ec55c2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:09.644 7f56ec55c2e8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:09.644 7f56ec55c2e8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:09.644 7f56ec55c2e8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:09.783 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table', created 03/24/2025 00:49:09.793 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:49:09.805 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock acquired. Performing schema change... 03/24/2025 00:49:10.729 7f56f65a6d08 [contrail-device-manager] [WARNING]: table 'dm_pnf_resource_table' fixed with properties {'gc_grace_seconds': 864000} 03/24/2025 00:49:10.735 7f56f65a6d08 [contrail-device-manager] [WARNING]: Zookeeper lock released. 03/24/2025 00:49:10.735 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.735 7f56ecf35cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:10.735 7f56ecf35cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.807 7f56f65a6d08 [contrail-device-manager] [WARNING]: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.23:5673//', 'pyamqp://guest:guest@10.0.0.50:5673//', 'pyamqp://guest:guest@10.0.0.57:5673//'] 03/24/2025 00:49:10.807 7f56f65a6d08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection down 03/24/2025 00:49:10.808 7f56ec554588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.808 7f56ec554588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Initializing description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 03/24/2025 00:49:10.808 7f56ec554588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.808 7f56ec554588 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.808 7f56ec554588 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 03/24/2025 00:49:10.808 7f56ec554588 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.829 7f56f65a6d08 [contrail-device-manager] [WARNING]: RabbitMQ drainer connection ESTABLISHED 03/24/2025 00:49:10.829 7f56ecf35630 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.829 7f56ecf35630 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:10.829 7f56ecf35630 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.885 7f56f65a6d08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 03/24/2025 00:49:10.885 7f56ec562048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.886 7f56ec562048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 03/24/2025 00:49:10.886 7f56ec562048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.888 7f56f65a6d08 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 03/24/2025 00:49:10.888 7f56ec562eb8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.888 7f56ec562eb8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:10.889 7f56ec562eb8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.889 7f56ec6cee10 [contrail-device-manager] [WARNING]: RabbitMQ publish connection down 03/24/2025 00:49:10.890 7f56ec728c18 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.890 7f56ec728c18 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Down description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 03/24/2025 00:49:10.890 7f56ec728c18 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.899 7f56ec6cee10 [contrail-device-manager] [WARNING]: RabbitMQ publish connection ESTABLISHED 03/24/2025 00:49:10.899 7f56ec562390 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:10.899 7f56ec562390 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:10.899 7f56ec562390 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:13.643 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:13.647 7f56ec5b6978 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:13.647 7f56ec5b6978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:13.647 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:13.647 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:13.647 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:13.647 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:13.648 7f56ec5b6978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:13.648 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:13.648 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:13.648 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:13.648 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:13.648 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:13.648 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:17.648 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:17.649 7f56ec5b6978 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:17.649 7f56ec5b6978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:17.649 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:17.649 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:17.650 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:17.650 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:17.650 7f56ec5b6978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:17.650 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:17.650 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:17.650 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:17.650 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:17.650 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:17.650 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:21.650 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:21.652 7f56ec5b6978 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:21.652 7f56ec5b6978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:21.652 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:21.652 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:21.652 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:21.652 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:21.653 7f56ec5b6978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:21.653 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:21.653 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:21.653 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:21.653 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:21.653 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:21.653 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:25.652 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:25.654 7f56ec5b6978 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:25.654 7f56ec5b6978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:25.654 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:25.654 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:25.654 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:25.654 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:25.655 7f56ec5b6978 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:25.655 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:25.655 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:25.655 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:25.655 7f56ec5b6978 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:25.655 7f56ec5b6978 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:25.655 7f56ec5b6978 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:28.289 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-2'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.6', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.6', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-2', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-2'], 'id_perms': {'created': '2025-03-24T00:49:28.276504', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:49:28.276504', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10073190237656315588, 'uuid_mslong': 13065924669758914641}}, 'parent_type': 'routing-instance', 'parent_uuid': '50085a56-a72a-4771-b0d3-2cfa384d5a33', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b5537d42-cafe-4051-8bcb-2926ae6ff6c4'}, 'oper': 'CREATE', 'request-id': 'req-a12c58ad-390a-4de1-affc-ee32c85888f2', 'type': 'bgp_router', 'uuid': 'b5537d42-cafe-4051-8bcb-2926ae6ff6c4'} 03/24/2025 00:49:28.322 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-3'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.21', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.21', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-3', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-3'], 'id_perms': {'created': '2025-03-24T00:49:28.312142', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:49:28.312142', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11360110580585673478, 'uuid_mslong': 8816493513183416122}}, 'parent_type': 'routing-instance', 'parent_uuid': '50085a56-a72a-4771-b0d3-2cfa384d5a33', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '7a5a7a23-71d5-4f3a-9da7-386274f73306'}, 'oper': 'CREATE', 'request-id': 'req-41edadaf-9832-4007-b465-af6f351c9c14', 'type': 'bgp_router', 'uuid': '7a5a7a23-71d5-4f3a-9da7-386274f73306'} 03/24/2025 00:49:28.358 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-1'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.5', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.5', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-1', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-1'], 'id_perms': {'created': '2025-03-24T00:49:28.345965', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:49:28.345965', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10178552049938250287, 'uuid_mslong': 6563749638263030775}}, 'parent_type': 'routing-instance', 'parent_uuid': '50085a56-a72a-4771-b0d3-2cfa384d5a33', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5b171f85-0f44-43f7-8d41-7b294064ba2f'}, 'oper': 'CREATE', 'request-id': 'req-68c02b54-a4d1-444b-a825-87621dba71f6', 'type': 'bgp_router', 'uuid': '5b171f85-0f44-43f7-8d41-7b294064ba2f'} 03/24/2025 00:49:28.549 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-7bfdd64c-9cc3-40b6-9377-d69236e148a6', 'type': 'global_system_config', 'uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6'} 03/24/2025 00:49:28.583 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-cf1e5440-683b-4834-aaf6-dc5ec2fab7b8', 'type': 'global_system_config', 'uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6'} 03/24/2025 00:49:28.606 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-0fbf4770-e1ab-427d-a13f-1e6b3e644fd4', 'type': 'global_system_config', 'uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6'} 03/24/2025 00:49:28.903 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-3'], 'oper': 'UPDATE', 'request-id': 'req-1bb0520a-9a92-4ea0-a0a1-1797bd34e56a', 'type': 'bgp_router', 'uuid': '7a5a7a23-71d5-4f3a-9da7-386274f73306'} 03/24/2025 00:49:28.917 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-2'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-1bb0520a-9a92-4ea0-a0a1-1797bd34e56a', 'type': 'bgp_router', 'uuid': 'b5537d42-cafe-4051-8bcb-2926ae6ff6c4'} 03/24/2025 00:49:29.006 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-1'], 'oper': 'UPDATE', 'request-id': 'req-ef73680c-30fc-4d91-b4d5-05ed1812daeb', 'type': 'bgp_router', 'uuid': '5b171f85-0f44-43f7-8d41-7b294064ba2f'} 03/24/2025 00:49:29.019 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-2'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-ef73680c-30fc-4d91-b4d5-05ed1812daeb', 'type': 'bgp_router', 'uuid': 'b5537d42-cafe-4051-8bcb-2926ae6ff6c4'} 03/24/2025 00:49:29.072 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-1'], 'oper': 'UPDATE', 'request-id': 'req-bbb1cde1-f8b5-4a98-9cea-ee49252a48b9', 'type': 'bgp_router', 'uuid': '5b171f85-0f44-43f7-8d41-7b294064ba2f'} 03/24/2025 00:49:29.086 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-2938-3'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-bbb1cde1-f8b5-4a98-9cea-ee49252a48b9', 'type': 'bgp_router', 'uuid': '7a5a7a23-71d5-4f3a-9da7-386274f73306'} 03/24/2025 00:49:29.654 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:29.655 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:29.655 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:29.655 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:29.656 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:29.656 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:29.656 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:29.656 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:29.656 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:29.656 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:29.656 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:29.656 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:29.656 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:29.656 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:33.657 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:33.658 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:33.659 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:33.659 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:33.659 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:33.659 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:33.659 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:33.659 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:33.659 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:33.660 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:33.660 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:33.660 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:33.660 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:33.660 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:37.660 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:37.661 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:37.662 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:37.662 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:37.662 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:37.662 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:37.662 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:37.663 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:37.663 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:37.663 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:37.663 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:37.663 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:37.663 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:37.663 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:41.662 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:41.663 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:41.664 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:41.664 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:41.664 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:41.664 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:41.664 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:41.664 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:41.664 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:41.664 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:41.665 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:41.665 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:41.665 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:41.665 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:45.665 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:45.666 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:45.667 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:45.667 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:45.667 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:45.667 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:45.667 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:45.667 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:45.667 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:45.667 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:45.667 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:45.667 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:45.668 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:45.668 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:49.668 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:49.669 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:49.669 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:49.670 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:49.670 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:49.670 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:49.670 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:49.670 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:49.670 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:49.671 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:49.671 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:49.671 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:49.671 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:49.671 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:53.670 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:53.671 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:53.672 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:53.672 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:53.672 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:53.672 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:53.672 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:53.673 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:53.673 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:53.673 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:53.673 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:53.673 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:53.673 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:53.673 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:57.673 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:49:57.675 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:49:57.675 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:49:57.675 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:57.675 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:49:57.675 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:49:57.675 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:49:57.675 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:49:57.675 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:57.676 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:49:57.676 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:57.676 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:49:57.676 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:49:57.676 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:01.676 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:01.677 7f56ecf35cc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:01.677 7f56ecf35cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:01.678 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:01.678 7f56ecf35cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:01.678 7f56ecf35cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:01.678 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:01.678 7f56ecf35cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:01.678 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:01.679 7f56ecf35cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:01.679 7f56ecf35cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:01.679 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:01.679 7f56ecf35cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:01.679 7f56ecf35cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:05.677 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:05.678 7f56ecf35cc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:05.679 7f56ecf35cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:05.679 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:05.679 7f56ecf35cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:05.679 7f56ecf35cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:05.679 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:05.679 7f56ecf35cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:05.679 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:05.679 7f56ecf35cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:05.680 7f56ecf35cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:05.680 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:05.680 7f56ecf35cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:05.680 7f56ecf35cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:09.679 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:09.681 7f56ecf35cc0 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:09.681 7f56ecf35cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:09.681 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:09.681 7f56ecf35cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:09.681 7f56ecf35cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:09.682 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:09.682 7f56ecf35cc0 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:09.682 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:09.682 7f56ecf35cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:09.682 7f56ecf35cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:09.682 7f56ecf35cc0 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:09.682 7f56ecf35cc0 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:09.682 7f56ecf35cc0 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:13.682 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:13.684 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:13.684 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:13.684 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:13.684 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:13.684 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:13.684 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:13.684 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:13.684 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:13.684 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:13.685 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:13.685 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:13.685 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:13.685 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:17.685 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:17.686 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:17.686 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:17.686 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:17.686 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:17.686 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:17.686 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:17.687 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:17.687 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:17.687 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:17.687 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:17.687 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:17.687 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:17.687 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:21.687 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:21.689 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:21.689 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:21.689 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:21.689 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:21.689 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:21.690 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:21.690 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:21.690 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:21.690 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:21.690 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:21.690 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:21.690 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:21.690 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:25.694 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:25.696 7f56ec624a20 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:25.696 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:25.696 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:25.696 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:25.696 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:25.696 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:25.696 7f56ec624a20 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:25.696 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:25.697 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:25.697 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:25.697 7f56ec624a20 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:25.697 7f56ec624a20 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:25.697 7f56ec624a20 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:26.019 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-1.'], 'obj_dict': {'database_node_ip_address': '10.0.0.23', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-1.'], 'id_perms': {'created': '2025-03-24T00:50:26.001379', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:50:26.001379', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10322133253866064734, 'uuid_mslong': 3220258073310480216}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '2cb0a7a7-f0c1-4758-8f3f-958160ef0b5e'}, 'oper': 'CREATE', 'request-id': 'req-90444c5c-5c67-49df-8e53-4d060c1d9f94', 'type': 'database_node', 'uuid': '2cb0a7a7-f0c1-4758-8f3f-958160ef0b5e'} 03/24/2025 00:50:26.065 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-3.'], 'obj_dict': {'database_node_ip_address': '10.0.0.57', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-3.'], 'id_perms': {'created': '2025-03-24T00:50:26.051170', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:50:26.051170', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12318961881320041735, 'uuid_mslong': 14775151738435683769}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'cd0be239-57f4-45b9-aaf5-be8ff9a8ad07'}, 'oper': 'CREATE', 'request-id': 'req-effe2fbe-1e9f-4f6e-a28b-ba8d5c97bb99', 'type': 'database_node', 'uuid': 'cd0be239-57f4-45b9-aaf5-be8ff9a8ad07'} 03/24/2025 00:50:26.081 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-2.'], 'obj_dict': {'database_node_ip_address': '10.0.0.50', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-2.'], 'id_perms': {'created': '2025-03-24T00:50:26.072404', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:50:26.072404', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12203353673661822660, 'uuid_mslong': 6506263888850668969}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5a4ae489-05e5-4da9-a95b-0582931236c4'}, 'oper': 'CREATE', 'request-id': 'req-0a804bd1-14ba-46ea-9ea0-a2899acfb186', 'type': 'database_node', 'uuid': '5a4ae489-05e5-4da9-a95b-0582931236c4'} 03/24/2025 00:50:29.697 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:29.698 7f56ec6fe048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:29.698 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:29.698 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:29.698 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:29.698 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:29.698 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:29.699 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:29.699 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:29.699 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:29.699 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:29.699 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:29.699 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:29.699 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:33.698 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:33.700 7f56ec6fe048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:33.700 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:33.700 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:33.700 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:33.700 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:33.700 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:33.701 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:33.701 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:33.701 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:33.701 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:33.701 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:33.701 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:33.701 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:37.701 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:37.702 7f56ec6fe048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:37.702 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:37.702 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:37.703 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:37.703 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:37.703 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:37.703 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:37.703 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:37.703 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:37.703 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:37.703 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:37.703 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:37.703 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:41.704 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:41.705 7f56ec6fe048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:41.705 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:41.706 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:41.706 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:41.706 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:41.706 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:41.706 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:41.706 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:41.706 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:41.706 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:41.706 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:41.707 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:41.707 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:45.707 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:45.708 7f56ec6fe048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:45.709 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:45.709 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:45.709 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:45.709 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:45.709 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:45.709 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:45.709 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:45.709 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:45.709 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:45.709 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:45.710 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:45.710 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:49.710 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:49.712 7f56ec6fe048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:49.712 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:49.712 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:49.712 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:49.712 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:49.712 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:49.713 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:49.713 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:49.713 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:49.713 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:49.713 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:49.713 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:49.713 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:53.714 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:53.714 7f56ec6fe048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:53.715 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:53.715 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:53.715 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:53.715 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:53.715 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:53.715 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:53.715 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:53.715 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:53.715 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:53.716 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:53.716 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:53.716 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:57.715 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:50:57.716 7f56ec6fe048 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:50:57.716 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:50:57.717 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:57.717 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:50:57.717 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:50:57.717 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:50:57.717 7f56ec6fe048 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:50:57.717 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:57.717 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:50:57.717 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:57.717 7f56ec6fe048 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:50:57.717 7f56ec6fe048 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:50:57.718 7f56ec6fe048 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:01.718 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:51:01.720 7f56ec7106d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:51:01.721 7f56ec7106d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:51:01.721 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:51:01.721 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:51:01.721 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:51:01.721 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:51:01.722 7f56ec7106d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:51:01.722 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:01.722 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:51:01.722 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:01.722 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:01.722 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:51:01.723 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:05.721 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:51:05.722 7f56ec7106d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:51:05.722 7f56ec7106d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:51:05.722 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:51:05.723 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:51:05.723 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:51:05.723 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:51:05.723 7f56ec7106d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:51:05.723 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:05.723 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:51:05.723 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:05.723 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:05.723 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.23:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:51:05.724 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:09.724 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:51:09.725 7f56ec7106d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:51:09.725 7f56ec7106d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:51:09.725 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:51:09.726 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:51:09.726 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:51:09.726 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:51:09.726 7f56ec7106d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:51:09.726 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:09.726 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.50:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:51:09.726 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:09.726 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:09.726 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.50:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:51:09.726 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:13.726 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:51:13.728 7f56ec7106d8 [contrail-device-manager] [ERROR]: Session Event: TCP Connect Fail 03/24/2025 00:51:13.728 7f56ec7106d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:51:13.728 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:51:13.728 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:51:13.728 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:51:13.728 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/24/2025 00:51:13.728 7f56ec7106d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/24/2025 00:51:13.728 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:13.729 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.57:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/24/2025 00:51:13.729 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:13.729 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:13.729 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1.:Config:contrail-device-manager:0 client_info = << status = Idle successful_connections = 0 pid = 136 http_port = 8096 start_time = 1742777257569153 collector_name = collector_ip = 10.0.0.57:8086 collector_list = [ 10.0.0.23:8086, 10.0.0.50:8086, 10.0.0.57:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 2 >> 03/24/2025 00:51:13.729 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/24/2025 00:51:16.014 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.23', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-1.'], 'id_perms': {'created': '2025-03-24T00:51:16.002936', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:51:16.002936', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13009506911024807647, 'uuid_mslong': 9252151936240862393}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '80663f2a-7f94-44b9-b48b-0d9bb69f7edf'}, 'oper': 'CREATE', 'request-id': 'req-d9c452b8-519a-420c-96ba-3fad02855b4e', 'type': 'analytics_node', 'uuid': '80663f2a-7f94-44b9-b48b-0d9bb69f7edf'} 03/24/2025 00:51:16.096 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-2.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.50', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-2.'], 'id_perms': {'created': '2025-03-24T00:51:16.088635', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:51:16.088635', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11654343355259065041, 'uuid_mslong': 10219957459111726470}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8dd49529-51f5-4d86-a1bc-8b88c4a692d1'}, 'oper': 'CREATE', 'request-id': 'req-5c11a9ba-cbc6-43bc-8598-310c2a56ec70', 'type': 'analytics_node', 'uuid': '8dd49529-51f5-4d86-a1bc-8b88c4a692d1'} 03/24/2025 00:51:16.543 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-3.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.57', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-3.'], 'id_perms': {'created': '2025-03-24T00:51:16.536340', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:51:16.536340', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12821909360380823386, 'uuid_mslong': 4309689799275266538}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3bcf1809-e9ed-41ea-b1f0-92a13fc1475a'}, 'oper': 'CREATE', 'request-id': 'req-c5120111-fdf8-4d8c-bd37-41f167d67896', 'type': 'analytics_node', 'uuid': '3bcf1809-e9ed-41ea-b1f0-92a13fc1475a'} 03/24/2025 00:51:17.729 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/24/2025 00:51:17.730 7f56ec7106d8 [contrail-device-manager] [INFO]: Session Event: TCP Connected 03/24/2025 00:51:17.730 7f56ec7106d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/24/2025 00:51:17.730 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:51:17.730 7f56ec7106d8 [contrail-device-manager] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-2938-1. process_status = [ << module_id = contrail-device-manager instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = Collector server_addrs = [ 10.0.0.23:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.23:5673, 10.0.0.50:5673, 10.0.0.57:5673, ] status = Up description = >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.23:9041, 10.0.0.50:9041, 10.0.0.57:9041, ] status = Up description = >>, << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.23:2181, 10.0.0.50:2181, 10.0.0.57:2181, ] status = Up description = >>, << type = ApiServer name = ApiServer server_addrs = [ 10.0.0.23,10.0.0.50,10.0.0.57:8082, ] status = Up description = ApiServer Connection State updated >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/24/2025 00:51:17.730 7f56ec7106d8 [contrail-device-manager] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/24/2025 00:51:17.730 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/24/2025 00:51:17.730 7f56ec7106d8 [contrail-device-manager] [DEBUG]: Send sandesh control message. uve type count # 19 03/24/2025 00:51:17.731 7f56ec7106d8 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/24/2025 00:51:17.731 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/24/2025 00:51:17.731 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/24/2025 00:51:17.731 7f56ee8f3f60 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/24/2025 00:51:17.732 7f56ee8f3f60 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/24/2025 00:51:27.390 7f56ec5ad2e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001016 03/24/2025 00:51:27.390 7f56ec5ad2e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/24/2025 00:51:27.391 7f56f0407b70 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/24/2025 00:51:27.391 7f56f0407b70 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 0 03/24/2025 00:51:27.391 7f56f0407b70 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:51:27.391 7f56f0407b70 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/24/2025 00:51:27.391 7f56f0407b70 [contrail-device-manager] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/24/2025 00:51:27.391 7f56ee822588 [contrail-device-manager] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/24/2025 00:51:27.392 7f56ee822588 [contrail-device-manager] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/24/2025 00:51:39.303 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.23', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-1.'], 'id_perms': {'created': '2025-03-24T00:51:39.294480', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:51:39.294480', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11944285739395123086, 'uuid_mslong': 15060496085992883903}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd101a165-72c4-46bf-a5c2-a098b069078e'}, 'oper': 'CREATE', 'request-id': 'req-a18ad294-8552-45a5-8466-cd6c6c377268', 'type': 'analytics_alarm_node', 'uuid': 'd101a165-72c4-46bf-a5c2-a098b069078e'} 03/24/2025 00:51:39.409 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-3.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.57', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-3.'], 'id_perms': {'created': '2025-03-24T00:51:39.398535', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:51:39.398535', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12519710083476334768, 'uuid_mslong': 4005005418331326287}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3794a335-6327-474f-adbe-f1fd187ba8b0'}, 'oper': 'CREATE', 'request-id': 'req-7a155516-5d08-4c75-baa5-5f8a992f050e', 'type': 'analytics_alarm_node', 'uuid': '3794a335-6327-474f-adbe-f1fd187ba8b0'} 03/24/2025 00:51:39.452 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-2.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.50', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-2.'], 'id_perms': {'created': '2025-03-24T00:51:39.432452', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:51:39.432452', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12241212273145639358, 'uuid_mslong': 950191725603538633}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0d2fc257-111d-46c9-a9e1-85b4a8ff71be'}, 'oper': 'CREATE', 'request-id': 'req-4540cbcd-7737-4aa9-953f-755bc0cff2e9', 'type': 'analytics_alarm_node', 'uuid': '0d2fc257-111d-46c9-a9e1-85b4a8ff71be'} 03/24/2025 00:51:40.874 7f56ec5ad2e8 [contrail-device-manager] [DEBUG]: Extracted sandesh length: 0000001297 03/24/2025 00:51:40.875 7f56ec5ad2e8 [contrail-device-manager] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/24/2025 00:51:40.875 7f56ec7106d8 [contrail-device-manager] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 03/24/2025 00:51:40.876 7f56ec7106d8 [contrail-device-manager] [DEBUG]: Number of uve types in sandesh control message is 2 03/24/2025 00:51:40.876 7f56ec7106d8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:51:40.876 7f56ec7106d8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/24/2025 00:51:53.226 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.23', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-1.'], 'id_perms': {'created': '2025-03-24T00:51:53.217148', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:51:53.217148', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9262869853112873469, 'uuid_mslong': 11936748245653012551}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a5a7d949-2b6a-4047-808c-530dfb77b1fd'}, 'oper': 'CREATE', 'request-id': 'req-302efb75-b52f-49f0-b5e6-fd6eeda6ebf5', 'type': 'analytics_snmp_node', 'uuid': 'a5a7d949-2b6a-4047-808c-530dfb77b1fd'} 03/24/2025 00:51:53.307 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.50', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-2.'], 'id_perms': {'created': '2025-03-24T00:51:53.299618', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:51:53.299618', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10990295241421232093, 'uuid_mslong': 12399067898761069280}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ac12568f-939a-42e0-9885-5f4b019417dd'}, 'oper': 'CREATE', 'request-id': 'req-e5ae200d-9f5a-4417-ad3f-1fdcab7d9ef7', 'type': 'analytics_snmp_node', 'uuid': 'ac12568f-939a-42e0-9885-5f4b019417dd'} 03/24/2025 00:51:53.335 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.57', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-2938-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-2938-3.'], 'id_perms': {'created': '2025-03-24T00:51:53.323237', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:51:53.323237', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12503222806699342770, 'uuid_mslong': 10953723098391595442}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '9803711f-679f-49b2-ad84-5ee5d4e07bb2'}, 'oper': 'CREATE', 'request-id': 'req-10e2331d-c382-4eb6-a17a-550f62c067a6', 'type': 'analytics_snmp_node', 'uuid': '9803711f-679f-49b2-ad84-5ee5d4e07bb2'} 03/24/2025 00:53:46.787 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-e69a5d34-384d-4b25-8439-a82d098c9ac5', 'type': 'global_system_config', 'uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6'} 03/24/2025 00:53:46.937 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-2938-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-2938-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-2938-2'], 'id_perms': {'created': '2025-03-24T00:53:46.925563', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:53:46.925563', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10570124726377580446, 'uuid_mslong': 4540706568353432234}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3f03d490-f3fd-42aa-92b0-a06f8b4f079e', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.17'}, 'oper': 'CREATE', 'request-id': 'req-5a7a2451-8ac9-4871-8c72-bb8b91f48d32', 'type': 'virtual_router', 'uuid': '3f03d490-f3fd-42aa-92b0-a06f8b4f079e'} 03/24/2025 00:53:47.311 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-2938-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-2938-2', 'vhost0'], 'id_perms': {'created': '2025-03-24T00:53:47.290645', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:53:47.290645', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9686829712537091169, 'uuid_mslong': 8272535897481888782}}, 'parent_type': 'virtual-router', 'parent_uuid': '3f03d490-f3fd-42aa-92b0-a06f8b4f079e', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '72cdf391-a913-440e-866e-884cef78fc61', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:72:cd:f3:91:a9']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '40b89d6c-6f2c-4155-8ecd-599c0381f464'}]}, 'oper': 'CREATE', 'request-id': 'req-65ab0794-622e-4f6f-8d12-8a10f67d2e19', 'type': 'virtual_machine_interface', 'uuid': '72cdf391-a913-440e-866e-884cef78fc61'} 03/24/2025 00:53:47.362 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-2938-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-65ab0794-622e-4f6f-8d12-8a10f67d2e19', 'type': 'virtual_machine_interface', 'uuid': '72cdf391-a913-440e-866e-884cef78fc61'} 03/24/2025 00:53:47.747 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-2938-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-2938-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-2938-1'], 'id_perms': {'created': '2025-03-24T00:53:47.734186', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:53:47.734186', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11484748125861271923, 'uuid_mslong': 5364495320038459916}}, 'parent_type': 'global-system-config', 'parent_uuid': 'e3cd32c4-56a7-4c68-ae12-bb80278a0ed6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4a728412-3e23-4a0c-9f62-059258cb5973', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.23'}, 'oper': 'CREATE', 'request-id': 'req-3652bf1e-ff9e-4c00-9d66-be1bcb1eac49', 'type': 'virtual_router', 'uuid': '4a728412-3e23-4a0c-9f62-059258cb5973'} 03/24/2025 00:53:47.903 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-25d0a203-fd86-40e6-a456-f78f780d58d6', 'type': 'access_control_list', 'uuid': '0b84bb60-a7e8-4561-a792-9df6440330fc'} 03/24/2025 00:53:48.137 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-2938-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-2938-1', 'vhost0'], 'id_perms': {'created': '2025-03-24T00:53:48.115591', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-24T00:53:48.115591', 'permissions': {'group': 'member', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10455309157077749159, 'uuid_mslong': 1341760019687886425}}, 'parent_type': 'virtual-router', 'parent_uuid': '4a728412-3e23-4a0c-9f62-059258cb5973', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '129ee3a1-0351-4e59-9118-b848a4820da7', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:12:9e:e3:a1:03']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '40b89d6c-6f2c-4155-8ecd-599c0381f464'}]}, 'oper': 'CREATE', 'request-id': 'req-9833e6f2-f767-4799-b1bd-d5b0a4fc1dc7', 'type': 'virtual_machine_interface', 'uuid': '129ee3a1-0351-4e59-9118-b848a4820da7'} 03/24/2025 00:53:48.229 7f56ec6ce630 [contrail-device-manager] [DEBUG]: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-2938-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-9833e6f2-f767-4799-b1bd-d5b0a4fc1dc7', 'type': 'virtual_machine_interface', 'uuid': '129ee3a1-0351-4e59-9118-b848a4820da7'} 03/24/2025 00:54:04.236 7f56ec6156d8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:54:16.389 7f56ecf356d8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:54:29.459 7f56ee8f3f60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:54:41.768 7f56ee7be4e0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:54:53.893 7f56ee7be4e0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:55:06.049 7f56ed97fc18 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:55:18.390 7f56ed97fc18 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:55:31.236 7f56ec777a20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:55:43.432 7f56ec6902e8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:55:55.548 7f56ec6902e8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:56:07.916 7f56f0407b70 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:56:20.146 7f56f0407b70 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:56:32.739 7f56ec55f630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:56:45.079 7f56ec55f588 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:56:57.334 7f56ec55f588 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:57:09.747 7f56ec635d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:57:22.107 7f56ec635d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:57:34.277 7f56f35ce780 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:57:46.534 7f56ec55c780 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:57:58.752 7f56ec55c780 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:58:10.991 7f56ecf35cc0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:58:23.218 7f56ecf35cc0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:58:35.481 7f56ed97fc18 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:58:47.760 7f56ec55c0f0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:59:00.020 7f56ec777a20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:59:12.420 7f56ec55fac8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:59:24.723 7f56ec55fac8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:59:37.066 7f56edceaf60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 00:59:49.326 7f56ec55c8d0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:00:01.511 7f56ec53ccc0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:00:15.170 7f56ec719a20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:00:27.541 7f56ec719a20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:00:40.220 7f56ec615d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:00:52.884 7f56ec615d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:01:05.195 7f56ed9b96d8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:01:17.482 7f56ec53f8d0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:01:29.865 7f56ec615978 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:01:42.908 7f56ec568780 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:01:55.063 7f56ec568780 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:02:07.365 7f56ec635d68 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:02:19.629 7f56ed97fc18 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:02:31.701 7f56ec55f8d0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:02:44.468 7f56ecf35cc0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:02:56.609 7f56ecf35cc0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:03:09.211 7f56ec5ad630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:03:21.497 7f56ec5ad630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:03:33.686 7f56ec710f60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:03:46.113 7f56ec5680f0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:03:58.313 7f56ec5680f0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:04:10.738 7f56ec6902e8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:04:23.114 7f56ec6902e8 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:04:35.373 7f56f35ce630 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:04:47.729 7f56ecf358d0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:05:00.289 7f56ec624a20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:05:12.553 7f56ec568e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:05:24.784 7f56ec568e10 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:05:36.924 7f56ec568f60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:05:49.227 7f56ec53f8d0 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:06:01.558 7f56f0407b70 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:06:13.925 7f56ec719a20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:06:26.289 7f56ec719a20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:06:28.751 7f56ec719a20 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:06:31.185 7f56ee822588 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/24/2025 01:07:03.903 7f56ee8f3f60 [contrail-device-manager] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map