09/11/2025 04:16:07.645 7fcca6de93c8 [contrail-api] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/11/2025 04:16:07.645 7fcca6de93c8 [contrail-api] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/11/2025 04:16:07.645 7fcca6de93c8 [contrail-api] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/11/2025 04:16:07.645 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to zookeeper on 10.0.0.254:2181,10.0.0.22:2181,10.0.0.48:2181 09/11/2025 04:16:07.646 7fcca6de93c8 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 09/11/2025 04:16:07.646 7fcc9dbc5e10 [contrail-api] [ERROR]: Cannot write http_port 8084 to /tmp/contrail-api.0.http_port 09/11/2025 04:16:07.646 7fcc9dbc5e10 [contrail-api] [ERROR]: Starting Introspect on HTTP Port 8084 09/11/2025 04:16:07.647 7fcc9dbcb780 [contrail-api] [INFO]: Processing event[EvStart] in state[none] 09/11/2025 04:16:07.647 7fcc9dbcb780 [contrail-api] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/11/2025 04:16:07.647 7fcc9dbcb780 [contrail-api] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/11/2025 04:16:07.648 7fcc9dbcb780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:07.648 7fcc9dbcb780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >> 09/11/2025 04:16:07.648 7fcc9dbcb780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:07.648 7fcc9dbcb780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:07.648 7fcc9dbcb780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/11/2025 04:16:07.648 7fcc9dbcb780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:07.648 7fcc9dbcb780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:07.648 7fcc9dbcb780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564167647624 collector_name = collector_ip = collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 09/11/2025 04:16:07.648 7fcc9dbcb780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:07.656 7fcc9dbf60f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:07.656 7fcc9dbf60f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/11/2025 04:16:07.656 7fcc9dbf60f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:11.652 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:16:11.653 7fcc9db948d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:16:11.653 7fcc9db948d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:16:11.654 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:11.654 7fcc9db948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:16:11.654 7fcc9db948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:11.654 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:16:11.655 7fcc9db948d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:16:11.655 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:11.655 7fcc9db948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:11.655 7fcc9db948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:11.655 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:11.655 7fcc9db948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564167647624 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:16:11.655 7fcc9db948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:15.658 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:16:15.659 7fcc9db948d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:16:15.659 7fcc9db948d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:16:15.659 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:15.659 7fcc9db948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:16:15.660 7fcc9db948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:15.660 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:16:15.660 7fcc9db948d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:16:15.661 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:15.661 7fcc9db948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:15.661 7fcc9db948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:15.661 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:15.661 7fcc9db948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564167647624 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:16:15.661 7fcc9db948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:19.661 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:16:19.663 7fcc9db948d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:16:19.663 7fcc9db948d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:16:19.663 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:19.663 7fcc9db948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:16:19.663 7fcc9db948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:19.663 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:16:19.663 7fcc9db948d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:16:19.663 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:19.664 7fcc9db948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:19.664 7fcc9db948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:19.664 7fcc9db948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:19.664 7fcc9db948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564167647624 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:16:19.664 7fcc9db948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.065 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to database on ['10.0.0.254:9041', '10.0.0.22:9041', '10.0.0.48:9041'] 09/11/2025 04:16:21.065 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: VNCCassandra started with driver cql 09/11/2025 04:16:21.065 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: initializing cassandra cluster 09/11/2025 04:16:21.065 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster before create 09/11/2025 04:16:21.067 7fcc9db94c18 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.067 7fcc9db94c18 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/11/2025 04:16:21.067 7fcc9db94c18 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.068 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster created 09/11/2025 04:16:21.108 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster DB connected 09/11/2025 04:16:21.109 7fcc9dbb08d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.109 7fcc9dbb08d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/11/2025 04:16:21.109 7fcc9dbb08d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.115 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster init KS 09/11/2025 04:16:21.116 7fcc9dbb06d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.117 7fcc9dbb06d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:Cassandra[] connection down >>, ] >> 09/11/2025 04:16:21.117 7fcc9dbb06d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.132 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster KS are ready 09/11/2025 04:16:21.188 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster CF are ready 09/11/2025 04:16:21.189 7fcc9da46438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.189 7fcc9da46438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:21.189 7fcc9da46438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.283 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.254:5673//', 'pyamqp://guest:guest@10.0.0.22:5673//', 'pyamqp://guest:guest@10.0.0.48:5673//'] 09/11/2025 04:16:21.291 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection down 09/11/2025 04:16:21.292 7fcc9d1b5630 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.292 7fcc9d1b5630 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/11/2025 04:16:21.293 7fcc9d1b5630 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.293 7fcc9d1b5630 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.293 7fcc9d1b5630 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/11/2025 04:16:21.293 7fcc9d1b5630 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.303 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection ESTABLISHED 09/11/2025 04:16:21.303 7fcc9d047828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.303 7fcc9d047828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:21.303 7fcc9d047828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.355 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/11/2025 04:16:21.356 7fcc9d1b58d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.356 7fcc9d1b58d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/11/2025 04:16:21.356 7fcc9d1b58d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.365 7fcca6de93c8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/11/2025 04:16:21.366 7fcc9d1b5198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.366 7fcc9d1b5198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:21.366 7fcc9d1b5198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.372 7fcc9d1b5780 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/11/2025 04:16:21.373 7fcc9d04b048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.373 7fcc9d04b048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Down description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Database:RabbitMQ[] connection down >>, ] >> 09/11/2025 04:16:21.373 7fcc9d04b048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.404 7fcc9d1b5780 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/11/2025 04:16:21.405 7fcc9d189588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:21.405 7fcc9d189588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:21.405 7fcc9d189588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.345 7fa02fc42588 [contrail-api] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 09/11/2025 04:16:31.345 7fa02fc42588 [contrail-api] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 09/11/2025 04:16:31.345 7fa02fc42588 [contrail-api] [INFO]: SANDESH: Flow Logging: [None] -> [False] 09/11/2025 04:16:31.345 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to zookeeper on 10.0.0.254:2181,10.0.0.22:2181,10.0.0.48:2181 09/11/2025 04:16:31.346 7fa02fc42588 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 09/11/2025 04:16:31.347 7fa026a53e10 [contrail-api] [ERROR]: Cannot write http_port 8084 to /tmp/contrail-api.0.http_port 09/11/2025 04:16:31.347 7fa026a53e10 [contrail-api] [ERROR]: Starting Introspect on HTTP Port 8084 09/11/2025 04:16:31.348 7fa0269e8780 [contrail-api] [INFO]: Processing event[EvStart] in state[none] 09/11/2025 04:16:31.348 7fa0269e8780 [contrail-api] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/11/2025 04:16:31.350 7fa0269e8780 [contrail-api] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 09/11/2025 04:16:31.350 7fa0269e8780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.350 7fa0269e8780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >> 09/11/2025 04:16:31.350 7fa0269e8780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.350 7fa0269e8780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.351 7fa0269e8780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/11/2025 04:16:31.351 7fa0269e8780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.351 7fa0269e8780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.351 7fa0269e8780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 09/11/2025 04:16:31.351 7fa0269e8780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.359 7fa026a100f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.360 7fa026a100f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/11/2025 04:16:31.360 7fa026a100f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.403 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to database on ['10.0.0.254:9041', '10.0.0.22:9041', '10.0.0.48:9041'] 09/11/2025 04:16:31.403 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: VNCCassandra started with driver cql 09/11/2025 04:16:31.403 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: initializing cassandra cluster 09/11/2025 04:16:31.404 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster before create 09/11/2025 04:16:31.404 7fa0269ac780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.404 7fa0269ac780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/11/2025 04:16:31.404 7fa0269ac780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.405 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster created 09/11/2025 04:16:31.433 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster DB connected 09/11/2025 04:16:31.436 7fa0269cd780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.436 7fa0269cd780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/11/2025 04:16:31.436 7fa0269cd780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.443 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster init KS 09/11/2025 04:16:31.443 7fa0299cd4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.443 7fa0299cd4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 09/11/2025 04:16:31.444 7fa0299cd4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.451 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster KS are ready 09/11/2025 04:16:31.501 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster CF are ready 09/11/2025 04:16:31.502 7fa02685f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.503 7fa02685f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/11/2025 04:16:31.503 7fa02685f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.600 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.254:5673//', 'pyamqp://guest:guest@10.0.0.22:5673//', 'pyamqp://guest:guest@10.0.0.48:5673//'] 09/11/2025 04:16:31.611 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection down 09/11/2025 04:16:31.612 7fa025f7b588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.612 7fa025f7b588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/11/2025 04:16:31.612 7fa025f7b588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.612 7fa025f7b588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.612 7fa025f7b588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/11/2025 04:16:31.612 7fa025f7b588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.616 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection ESTABLISHED 09/11/2025 04:16:31.616 7fa025ec6e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.616 7fa025ec6e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/11/2025 04:16:31.616 7fa025ec6e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.676 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/11/2025 04:16:31.676 7fa02685cd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.677 7fa02685cd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/11/2025 04:16:31.677 7fa02685cd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.680 7fa02fc42588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/11/2025 04:16:31.681 7fa025eca048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.681 7fa025eca048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/11/2025 04:16:31.681 7fa025eca048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.682 7fa025f7b588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 09/11/2025 04:16:31.683 7fa025f3f048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.683 7fa025f3f048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Down description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:RabbitMQ[] connection down >>, ] >> 09/11/2025 04:16:31.684 7fa025f3f048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.688 7fa025f7b588 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 09/11/2025 04:16:31.688 7fa025f3fa20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.688 7fa025f3fa20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 09/11/2025 04:16:31.688 7fa025f3fa20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:31.935 7fa02689a390 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Executing db_resync on api-server instance 10.0.0.22_0 09/11/2025 04:16:31.936 7fa02689a390 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Contrail Version is(nightly-r24.1-centos) 09/11/2025 04:16:31.944 7fa02689a390 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: zk_sync_node data (nightly-r24.1-centos) 09/11/2025 04:16:31.947 7fa02689a390 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DBERESYNC: running walk 09/11/2025 04:16:31.967 7fa02689a390 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Cassandra DB walk completed. 09/11/2025 04:16:31.993 7fa02689a390 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Time elapsed in resyncing db: 0:00:00.056211 09/11/2025 04:16:32.002 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'oper': 'UPDATE', 'request-id': 'req-e70ed867-3d9f-4a20-bdd9-047800c3ab93', 'type': 'application_policy_set', 'uuid': '60f9f9d6-33f1-498f-bb95-2a1a3151b512'} 09/11/2025 04:16:32.012 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 09/11/2025 04:16:32.015 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'oper': 'UPDATE', 'request-id': 'req-42bb673a-5af1-4a1b-aec7-2632ba2cde07', 'type': 'network_ipam', 'uuid': '2cc405b2-832e-42c7-9f5a-0fbca3cf48d0'} 09/11/2025 04:16:32.017 7fa02d628f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:32.018 7fa02d628f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-policy-management:default-application-policy-set elements = { all_applications : truefq_name : ["default-policy-management", "default-application-policy-set"]global_vrouter_config_refs : [{"to": ["default-global-system-config", "default-global-vrouter-config"], "attr": null, "uuid": "b6b25520-beae-4e43-885e-8e9d517df70c"}]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6987890995231803791, "uuid_lslong": 13516756148607366418}, "enable": true, "created": "2025-09-11T04:16:21.671952", "last_modified": "2025-09-11T04:16:31.771432", "description": null, "user_visible": true, "creator": null}parent_type : "policy-management"parent_uuid : "44d63150-6907-438f-96ea-1a3ba3a7e9cc"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "60f9f9d6-33f1-498f-bb95-2a1a3151b512" } deleted = False >> 09/11/2025 04:16:32.018 7fa02d628f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:32.027 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'oper': 'UPDATE', 'request-id': 'req-39156012-e23a-4f10-a626-07c59a278b8e', 'type': 'project', 'uuid': '6b83acaf-67eb-4c3b-aba7-8727560a43c6'} 09/11/2025 04:16:32.033 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectProjectTable:ContrailConfigTrace] map 09/11/2025 04:16:32.034 7fa02685cd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:32.034 7fa02685cd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-domain:default-project elements = { fq_name : ["default-domain", "default-project"]id_perms : {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7747225653370113083, "uuid_lslong": 12369003504613344198}, "enable": true, "created": "2025-09-11T04:16:21.451462", "last_modified": "2025-09-11T04:16:31.981356", "description": null, "user_visible": true, "creator": null}parent_type : "domain"parent_uuid : "7134262f-1ffb-453e-aad9-fbdd9a7a2b72"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}quota : {"defaults": -1}uuid : "6b83acaf-67eb-4c3b-aba7-8727560a43c6" } deleted = False >> 09/11/2025 04:16:32.034 7fa02685cd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:32.268 7fa02fc42588 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ModuleCpuStateTrace] map 09/11/2025 04:16:32.269 7fa02d2666d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:32.270 7fa02d2666d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ModuleCpuStateTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. build_info = {"build-info" : [{"build-version" : "v0+nightly.r24.1.centos", "build-time" : "2025-09-11 00:11:28.695100", "build-user" : "root", "build-hostname" : "jenkins-build-prepare-sandbox-10925-1.novalocal", "build-id" : "1", "build-number" : "centos"}]} config_node_ip = [ 10.0.0.22, 10.20.0.254, 172.17.0.1, ] >> 09/11/2025 04:16:32.270 7fa02d2666d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:32.398 7fa02603be10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:32.398 7fa02603be10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Generic Connection:Keystone[] connection down >>, ] >> 09/11/2025 04:16:32.399 7fa02603be10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:32.418 7fa02689aeb8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: KombuAmqpClient: Starting consumer greenlet 09/11/2025 04:16:32.418 7fa02689aeb8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: KombuAmqpClient: Ensuring Consumer connection 09/11/2025 04:16:32.419 7fa02689a6d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: KombuAmqpClient: Starting publisher greenlet 09/11/2025 04:16:32.419 7fa02689a6d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: KombuAmqpClient: Ensuring Publisher connection 09/11/2025 04:16:32.420 7fa02fc42588 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: KombuAmqpClient: Added consumer: amqp_request.cn-jenkins-deploy-platform-ansible-os-4080-2..dummy 09/11/2025 04:16:32.420 7fa02fc42588 [contrail-api] [DEBUG]: Add uve in the [none:ConfigApiWorkerTrace] map 09/11/2025 04:16:32.421 7fa02816c588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:32.422 7fa02816c588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ConfigApiWorkerTrace: data = << name = worker-0 worker_id = 0 introspect_port = 8084 admin_port = 8095 introspect_url = http://10.0.0.22:8084 >> 09/11/2025 04:16:32.422 7fa02816c588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:32.445 7fa02689a6d8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Publisher connection established 09/11/2025 04:16:32.446 7fa02689aeb8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/11/2025 04:16:32.508 7fa02689aeb8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-4080-2..dummy'] 09/11/2025 04:16:32.531 7fa02689aeb8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: KombuAmqpClient: Ensuring Consumer connection 09/11/2025 04:16:32.536 7fa02689aeb8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 09/11/2025 04:16:32.614 7fa02689aeb8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-ansible-os-4080-2..dummy'] 09/11/2025 04:16:34.602 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-b03e0c45-ff6b-43dc-a182-1d192ba1b9f6', 'type': 'global_vrouter_config', 'uuid': 'b6b25520-beae-4e43-885e-8e9d517df70c'} 09/11/2025 04:16:34.610 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-a115a8c5-f291-453f-a9fd-c792666d060c', 'type': 'global_vrouter_config', 'uuid': 'b6b25520-beae-4e43-885e-8e9d517df70c'} 09/11/2025 04:16:35.349 7fa025a948d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:16:35.351 7fa025a948d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:16:35.351 7fa025a948d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:16:35.351 7fa025a948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:35.351 7fa025a948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Generic Connection:Keystone[] connection down >>, ] >> 09/11/2025 04:16:35.351 7fa025a948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:35.351 7fa025a948d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:16:35.352 7fa025a948d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:16:35.352 7fa025a948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:35.352 7fa025a948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Initializing description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail], Generic Connection:Keystone[] connection down >>, ] >> 09/11/2025 04:16:35.352 7fa025a948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:35.352 7fa025a948d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:35.352 7fa025a948d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:16:35.352 7fa025a948d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:36.112 7fa025ad9588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:36.112 7fa025ad9588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:36.112 7fa025ad9588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:36.315 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-f640c775-6809-4957-b455-416c254931c1', 'type': 'global_vrouter_config', 'uuid': 'b6b25520-beae-4e43-885e-8e9d517df70c'} 09/11/2025 04:16:38.035 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements', 'operand2': {'json_value': '1'}, 'operation': 'size=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-elements-inconsistency', 'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'id_perms': {'created': '2025-09-11T04:16:38.024588', 'creator': None, 'description': 'ContrailConfig has a list of ' 'elements - it should be a dict. ' 'Controllers have inconsistency in ' 'their data.', 'enable': True, 'last_modified': '2025-09-11T04:16:38.024588', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9375909152530432018, 'uuid_mslong': 11570650112545933694}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a09334f9-32a2-497e-821d-ebb2d2e90012', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-58a11cc2-c551-4d7d-929e-67e309fdf5d0', 'type': 'alarm', 'uuid': 'a09334f9-32a2-497e-821d-ebb2d2e90012'} 09/11/2025 04:16:38.082 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_ip_address', 'operand2': {'uve_attribute': 'VrouterAgent.control_ip'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-compute', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'id_perms': {'created': '2025-09-11T04:16:38.075768', 'creator': None, 'description': 'Compute Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-11T04:16:38.075768', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12849629455095365658, 'uuid_mslong': 9279460659080611217}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '80c7444e-03b6-4591-b253-0de8be5b001a', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-efc34d43-134c-4e0f-82c7-2a9266998d51', 'type': 'alarm', 'uuid': '80c7444e-03b6-4591-b253-0de8be5b001a'} 09/11/2025 04:16:38.164 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.bgp_router_parameters.address', 'operand2': {'uve_attribute': 'BgpRouterState.bgp_router_ip_list'}, 'operation': 'not ' 'in'}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-control', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'id_perms': {'created': '2025-09-11T04:16:38.157671', 'creator': None, 'description': 'Control Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-09-11T04:16:38.157671', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13572203850105146895, 'uuid_mslong': 4717301548610832269}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '417738c3-df59-478d-bc5a-277e38b4ce0f', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-5cac5a50-9744-4b07-b8b3-4d3833893fc3', 'type': 'alarm', 'uuid': '417738c3-df59-478d-bc5a-277e38b4ce0f'} 09/11/2025 04:16:38.247 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_bgp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-bgp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'id_perms': {'created': '2025-09-11T04:16:38.239706', 'creator': None, 'description': 'BGP peer mismatch. Not enough BGP ' 'peers are up.', 'enable': True, 'last_modified': '2025-09-11T04:16:38.239706', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9658663651386035181, 'uuid_mslong': 10768107078987042227}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '95700056-2264-4db3-860a-776ac9d78bed', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-f479ea7d-894e-4b7f-adfe-3d731f1e6f00', 'type': 'alarm', 'uuid': '95700056-2264-4db3-860a-776ac9d78bed'} 09/11/2025 04:16:38.358 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-incorrect', 'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'id_perms': {'created': '2025-09-11T04:16:38.348762', 'creator': None, 'description': 'ContrailConfig missing or ' 'incorrect. Configuration pushed to ' 'Ifmap as ContrailConfig is ' 'missing/incorrect.', 'enable': True, 'last_modified': '2025-09-11T04:16:38.348762', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12260259400501729535, 'uuid_mslong': 14800930821642996115}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'cd677829-e129-4993-aa25-30f7950c98ff', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-2610d589-6e02-4544-ab7f-d9a87c1ce28e', 'type': 'alarm', 'uuid': 'cd677829-e129-4993-aa25-30f7950c98ff'} 09/11/2025 04:16:38.460 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '[70, ' '90]'}, 'operation': 'range', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-disk-usage-high', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'id_perms': {'created': '2025-09-11T04:16:38.452141', 'creator': None, 'description': 'Disk usage crosses high threshold ' 'limit.', 'enable': True, 'last_modified': '2025-09-11T04:16:38.452141', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10308463152618424312, 'uuid_mslong': 13993614131514722752}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c2334de9-48d2-49c0-8f0f-049ee80647f8', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-20ab4859-633f-43ce-836f-94fc6e5d923f', 'type': 'alarm', 'uuid': 'c2334de9-48d2-49c0-8f0f-049ee80647f8'} 09/11/2025 04:16:38.569 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '90'}, 'operation': '>', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-disk-usage-critical', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'id_perms': {'created': '2025-09-11T04:16:38.557466', 'creator': None, 'description': 'Disk usage crosses critical ' 'threshold limit.', 'enable': True, 'last_modified': '2025-09-11T04:16:38.557466', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9877567185324685666, 'uuid_mslong': 6106706195592135118}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '54bf60ee-1ff2-41ce-8914-2b057e553962', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-3b34bc6a-0c82-455d-b819-7787b3b27010', 'type': 'alarm', 'uuid': '54bf60ee-1ff2-41ce-8914-2b057e553962'} 09/11/2025 04:16:38.664 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-node-status', 'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'id_perms': {'created': '2025-09-11T04:16:38.653926', 'creator': None, 'description': 'Node Failure. NodeStatus UVE not ' 'present.', 'enable': True, 'last_modified': '2025-09-11T04:16:38.653926', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10696405863906203653, 'uuid_mslong': 6139249840589261399}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5532ff33-efb2-4a57-9471-44761388d805', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-a04f6724-a547-4d00-849c-4cffe53179b9', 'type': 'alarm', 'uuid': '5532ff33-efb2-4a57-9471-44761388d805'} 09/11/2025 04:16:38.755 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.build_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-partial-sysinfo', 'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'id_perms': {'created': '2025-09-11T04:16:38.741392', 'creator': None, 'description': 'System Info Incomplete.', 'enable': True, 'last_modified': '2025-09-11T04:16:38.741392', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10446616369821906719, 'uuid_mslong': 53935230771938060}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '00bf9dce-7f0a-430c-90f9-d63d6403f71f', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-32d53baf-7d88-4b65-a866-22fc08d9a6fd', 'type': 'alarm', 'uuid': '00bf9dce-7f0a-430c-90f9-d63d6403f71f'} 09/11/2025 04:16:38.850 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_status', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_status.state', 'operand2': {'json_value': '"Functional"'}, 'operation': '!=', 'variables': ['NodeStatus.process_status.module_id', 'NodeStatus.process_status.instance_id']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'id_perms': {'created': '2025-09-11T04:16:38.834810', 'creator': None, 'description': 'Process(es) reporting as ' 'non-functional.', 'enable': True, 'last_modified': '2025-09-11T04:16:38.834810', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13643699090125121779, 'uuid_mslong': 15228796985064966781}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd3578e2c-1b56-4a7d-bd58-2808f93a68f3', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-3bc64d9e-4df5-4b67-91ad-ad6e39158f36', 'type': 'alarm', 'uuid': 'd3578e2c-1b56-4a7d-bd58-2808f93a68f3'} 09/11/2025 04:16:38.933 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_info.process_state', 'operand2': {'json_value': '"PROCESS_STATE_RUNNING"'}, 'operation': '!=', 'variables': ['NodeStatus.process_info.process_name']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-status', 'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'id_perms': {'created': '2025-09-11T04:16:38.925975', 'creator': None, 'description': 'Process Failure.', 'enable': True, 'last_modified': '2025-09-11T04:16:38.925975', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9769008626938123312, 'uuid_mslong': 17379684384780275321}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f1310add-cf3b-4e79-8792-7d96785f9430', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-1d73457a-f81b-4f65-9da0-f24dd6b9f9b0', 'type': 'alarm', 'uuid': 'f1310add-cf3b-4e79-8792-7d96785f9430'} 09/11/2025 04:16:39.013 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.connected_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'id_perms': {'created': '2025-09-11T04:16:39.005199', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tor agent does not exist we look ' 'for non-empty value for ' 'connected_agent_list', 'enable': True, 'last_modified': '2025-09-11T04:16:39.005199', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9526402570379447079, 'uuid_mslong': 9476497755146701894}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8383487f-ea4e-4c46-8434-94aa2ad5cb27', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-4e8b9a86-7049-43e9-89b5-e8b9091dbdeb', 'type': 'alarm', 'uuid': '8383487f-ea4e-4c46-8434-94aa2ad5cb27'} 09/11/2025 04:16:39.099 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '!='}]}, {'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-tsn-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'id_perms': {'created': '2025-09-11T04:16:39.090644', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tsn agent does not exist we look ' 'for non-empty value for ' 'tsn_agent_list', 'enable': True, 'last_modified': '2025-09-11T04:16:39.090644', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10864495468544117286, 'uuid_mslong': 4768088288710640787}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '422ba709-424c-4493-96c6-7110ea69ca26', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-5f70e48c-be35-425d-a2e9-1fcdbb8d0ec0', 'type': 'alarm', 'uuid': '422ba709-424c-4493-96c6-7110ea69ca26'} 09/11/2025 04:16:39.198 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'StorageCluster.info_stats.status', 'operand2': {'json_value': '0'}, 'operation': '!=', 'variables': ['StorageCluster.info_stats.health_summary']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-storage-cluster-state', 'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'id_perms': {'created': '2025-09-11T04:16:39.186691', 'creator': None, 'description': 'Storage Cluster warning/errors.', 'enable': True, 'last_modified': '2025-09-11T04:16:39.186691', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12959106328063405152, 'uuid_mslong': 671525865667972445}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0951bd37-88a0-4d5d-b3d7-fe8b8abc6c60', 'uve_keys': {'uve_key': ['storage-cluster']}}, 'oper': 'CREATE', 'request-id': 'req-a6afd93a-9802-456a-8aa0-9cb4182c6102', 'type': 'alarm', 'uuid': '0951bd37-88a0-4d5d-b3d7-fe8b8abc6c60'} 09/11/2025 04:16:39.291 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.down_interface_count', 'operand2': {'json_value': '1'}, 'operation': '>=', 'variables': ['VrouterAgent.error_intf_list', 'VrouterAgent.no_config_intf_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-interface', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'id_perms': {'created': '2025-09-11T04:16:39.281454', 'creator': None, 'description': 'Vrouter interface(s) down.', 'enable': True, 'last_modified': '2025-09-11T04:16:39.281454', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10397870190407643300, 'uuid_mslong': 13911161745278583739}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c10e5fe7-2055-43bb-904c-a7d90fdef0a4', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-93250f55-e2cb-400f-a3e4-056540fb4b46', 'type': 'alarm', 'uuid': 'c10e5fe7-2055-43bb-904c-a7d90fdef0a4'} 09/11/2025 04:16:39.351 7fa025f7bb70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:16:39.352 7fa025f7bb70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:16:39.353 7fa025f7bb70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:16:39.353 7fa025f7bb70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:39.353 7fa025f7bb70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:16:39.353 7fa025f7bb70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:39.353 7fa025f7bb70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:16:39.353 7fa025f7bb70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:16:39.353 7fa025f7bb70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:39.354 7fa025f7bb70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:39.354 7fa025f7bb70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:39.354 7fa025f7bb70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:39.354 7fa025f7bb70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:16:39.354 7fa025f7bb70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:39.384 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_xmpp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'id_perms': {'created': '2025-09-11T04:16:39.370483', 'creator': None, 'description': 'XMPP peer mismatch.', 'enable': True, 'last_modified': '2025-09-11T04:16:39.370483', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10760828149662060548, 'uuid_mslong': 17344371186406278350}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f0b395b3-1a44-44ce-9556-243055a19004', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-c05c38c6-9e90-4806-8124-c53db61b295e', 'type': 'alarm', 'uuid': 'f0b395b3-1a44-44ce-9556-243055a19004'} 09/11/2025 04:16:39.479 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'XmppPeerInfoData.close_reason', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'XmppPeerInfoData.state_info.state', 'operand2': {'json_value': '"Established"'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-close-reason', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'id_perms': {'created': '2025-09-11T04:16:39.470256', 'creator': None, 'description': 'XMPP connection closed towards ' 'peer, ' 'alarm has close reason', 'enable': True, 'last_modified': '2025-09-11T04:16:39.470256', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11265760388192564603, 'uuid_mslong': 7268256497274406051}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '64de08c6-8077-44a3-9c58-05626ee2757b', 'uve_keys': {'uve_key': ['xmpp-peer']}}, 'oper': 'CREATE', 'request-id': 'req-5b0c2f1a-12a1-43f7-af4b-ee2543b7323d', 'type': 'alarm', 'uuid': '64de08c6-8077-44a3-9c58-05626ee2757b'} 09/11/2025 04:16:39.590 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-core-files', 'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'id_perms': {'created': '2025-09-11T04:16:39.573922', 'creator': None, 'description': 'A core file has been generated on ' 'the node.', 'enable': True, 'last_modified': '2025-09-11T04:16:39.573922', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12291247354303567403, 'uuid_mslong': 1068149691155499832}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0ed2d47c-1c44-4b38-aa93-4858cfb9562b', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-273ccc53-8a34-478e-8ec2-66b875e111cb', 'type': 'alarm', 'uuid': '0ed2d47c-1c44-4b38-aa93-4858cfb9562b'} 09/11/2025 04:16:39.667 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks', 'operand2': {'json_value': '300'}, 'operation': '>='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-pending-cassandra-compaction-tasks', 'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'id_perms': {'created': '2025-09-11T04:16:39.660640', 'creator': None, 'description': 'Pending compaction tasks in ' 'cassandra crossed the configured ' 'threshold.', 'enable': True, 'last_modified': '2025-09-11T04:16:39.660640', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11204195129837918366, 'uuid_mslong': 8205564413264087404}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '71e0055b-e231-496c-9b7d-4c1bb426989e', 'uve_keys': {'uve_key': ['database-nodeconfig-database-node']}}, 'oper': 'CREATE', 'request-id': 'req-f647c460-5333-4fd8-9f65-d07bded95245', 'type': 'alarm', 'uuid': '71e0055b-e231-496c-9b7d-4c1bb426989e'} 09/11/2025 04:16:39.749 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.running_package_version', 'operand2': {'uve_attribute': 'NodeStatus.installed_package_version'}, 'operation': '!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-package-version-mismatch', 'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'id_perms': {'created': '2025-09-11T04:16:39.742548', 'creator': None, 'description': 'There is a mismatch between ' 'installed and running package ' 'version.', 'enable': True, 'last_modified': '2025-09-11T04:16:39.742548', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10983564222402031793, 'uuid_mslong': 3723904913721804948}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '33adf7c9-f7b8-4894-986d-757783f670b1', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-20997d7c-12b8-4533-8819-2d016e828980', 'type': 'alarm', 'uuid': '33adf7c9-f7b8-4894-986d-757783f670b1'} 09/11/2025 04:16:39.831 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'id_perms': {'created': '2025-09-11T04:16:39.823746', 'creator': None, 'description': 'Agent resource usage exceeded ' 'configured watermark for resource.', 'enable': True, 'last_modified': '2025-09-11T04:16:39.823746', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10732038363437751285, 'uuid_mslong': 3586234090595568818}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '31c4dce7-9b58-4cb2-94ef-dc07e0925bf5', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-f3fc37a3-28d6-4ff1-b1fb-22a3777d9813', 'type': 'alarm', 'uuid': '31c4dce7-9b58-4cb2-94ef-dc07e0925bf5'} 09/11/2025 04:16:39.933 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_table_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-table-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'id_perms': {'created': '2025-09-11T04:16:39.920746', 'creator': None, 'description': 'Agent resource usage exceeded table ' 'size for resource in vrouter.', 'enable': True, 'last_modified': '2025-09-11T04:16:39.920746', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10273913627326672796, 'uuid_mslong': 3846134834198168038}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '35603741-06a3-41e6-8e94-4602b1d9ef9c', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-2bcc5bb1-9706-4cbb-b368-8b5d7ca00a86', 'type': 'alarm', 'uuid': '35603741-06a3-41e6-8e94-4602b1d9ef9c'} 09/11/2025 04:16:40.012 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.l3mh_phy_interface_down', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_l3mh_intf_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-l3mh-physical-interface-link-down', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'id_perms': {'created': '2025-09-11T04:16:40.005120', 'creator': None, 'description': 'On l3 multihomed compute one or ' 'more physical interfaces are ' 'Inactive.', 'enable': True, 'last_modified': '2025-09-11T04:16:40.005120', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12576893358341099760, 'uuid_mslong': 13867391756456838557}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c072df54-74ca-459d-ae8a-19dfc9e410f0', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-771c846c-1cbd-48ab-bec0-994a08b8b574', 'type': 'alarm', 'uuid': 'c072df54-74ca-459d-ae8a-19dfc9e410f0'} 09/11/2025 04:16:40.191 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'oper': 'UPDATE', 'request-id': 'req-e06532b7-0da0-4038-a9ce-77a639b60335', 'type': 'alarm', 'uuid': 'a09334f9-32a2-497e-821d-ebb2d2e90012'} 09/11/2025 04:16:40.363 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'oper': 'UPDATE', 'request-id': 'req-eca13644-648b-430d-a08c-683890c29d1d', 'type': 'alarm', 'uuid': '80c7444e-03b6-4591-b253-0de8be5b001a'} 09/11/2025 04:16:40.547 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'oper': 'UPDATE', 'request-id': 'req-970daa96-fa24-42f8-9599-3eb1ff07bc47', 'type': 'alarm', 'uuid': '417738c3-df59-478d-bc5a-277e38b4ce0f'} 09/11/2025 04:16:40.724 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-54c8996c-bd62-4f2d-9abd-e9fd7ab1990f', 'type': 'alarm', 'uuid': '95700056-2264-4db3-860a-776ac9d78bed'} 09/11/2025 04:16:40.883 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'oper': 'UPDATE', 'request-id': 'req-4d96ede4-a363-49ef-8794-a7c04d6840da', 'type': 'alarm', 'uuid': 'cd677829-e129-4993-aa25-30f7950c98ff'} 09/11/2025 04:16:41.071 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'oper': 'UPDATE', 'request-id': 'req-135e8482-bdf0-4797-8010-e88a9939f2dd', 'type': 'alarm', 'uuid': 'c2334de9-48d2-49c0-8f0f-049ee80647f8'} 09/11/2025 04:16:41.237 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'oper': 'UPDATE', 'request-id': 'req-49f4c19b-3a06-4d30-9a75-bfff9c727a2c', 'type': 'alarm', 'uuid': '54bf60ee-1ff2-41ce-8914-2b057e553962'} 09/11/2025 04:16:41.408 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'oper': 'UPDATE', 'request-id': 'req-82685397-4516-4622-b662-0cfa66b2db5a', 'type': 'alarm', 'uuid': '5532ff33-efb2-4a57-9471-44761388d805'} 09/11/2025 04:16:41.592 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'oper': 'UPDATE', 'request-id': 'req-08105351-8cd1-4b6e-a190-92cd712d0102', 'type': 'alarm', 'uuid': '00bf9dce-7f0a-430c-90f9-d63d6403f71f'} 09/11/2025 04:16:41.763 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-b2250e3a-de80-4b87-b329-22c31fc97704', 'type': 'alarm', 'uuid': 'd3578e2c-1b56-4a7d-bd58-2808f93a68f3'} 09/11/2025 04:16:41.932 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'oper': 'UPDATE', 'request-id': 'req-a102b5b9-5a1f-4f0c-ac4f-a1bc2ebf6f47', 'type': 'alarm', 'uuid': 'f1310add-cf3b-4e79-8792-7d96785f9430'} 09/11/2025 04:16:42.100 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-c94bc1d9-dc26-4a6e-a816-d7440aa3f402', 'type': 'alarm', 'uuid': '8383487f-ea4e-4c46-8434-94aa2ad5cb27'} 09/11/2025 04:16:42.275 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-0f0525b7-727a-489a-b3fd-c8bf22ee67e5', 'type': 'alarm', 'uuid': '422ba709-424c-4493-96c6-7110ea69ca26'} 09/11/2025 04:16:42.453 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'oper': 'UPDATE', 'request-id': 'req-3e44f62f-ef5f-43ee-9fc3-984d5f62e1e9', 'type': 'alarm', 'uuid': '0951bd37-88a0-4d5d-b3d7-fe8b8abc6c60'} 09/11/2025 04:16:42.636 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'oper': 'UPDATE', 'request-id': 'req-8654ed6d-e4fa-4a91-9576-35d2875ccd0e', 'type': 'alarm', 'uuid': 'c10e5fe7-2055-43bb-904c-a7d90fdef0a4'} 09/11/2025 04:16:42.834 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-8ba24df0-7848-4884-aa77-c305cfeccd6e', 'type': 'alarm', 'uuid': 'f0b395b3-1a44-44ce-9556-243055a19004'} 09/11/2025 04:16:42.999 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'oper': 'UPDATE', 'request-id': 'req-a4204585-7562-47aa-b960-edcaa865c7ed', 'type': 'alarm', 'uuid': '64de08c6-8077-44a3-9c58-05626ee2757b'} 09/11/2025 04:16:43.166 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'oper': 'UPDATE', 'request-id': 'req-94f091a7-c43c-4acd-b5df-ab816a8612d2', 'type': 'alarm', 'uuid': '0ed2d47c-1c44-4b38-aa93-4858cfb9562b'} 09/11/2025 04:16:43.334 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'oper': 'UPDATE', 'request-id': 'req-6d498398-11df-4000-a68d-c35c20d4cf78', 'type': 'alarm', 'uuid': '71e0055b-e231-496c-9b7d-4c1bb426989e'} 09/11/2025 04:16:43.353 7fa025bd92e8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:16:43.354 7fa025bd92e8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:16:43.355 7fa025bd92e8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:16:43.355 7fa025bd92e8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:43.355 7fa025bd92e8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:16:43.355 7fa025bd92e8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:43.355 7fa025bd92e8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:16:43.356 7fa025bd92e8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:16:43.356 7fa025bd92e8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:43.356 7fa025bd92e8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:43.356 7fa025bd92e8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:43.356 7fa025bd92e8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:43.356 7fa025bd92e8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:16:43.356 7fa025bd92e8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:43.508 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'oper': 'UPDATE', 'request-id': 'req-fae42891-1707-4a82-8f6c-20689a3d43f8', 'type': 'alarm', 'uuid': '33adf7c9-f7b8-4894-986d-757783f670b1'} 09/11/2025 04:16:43.674 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-526a36a3-e8c9-49d8-90d9-60b5e2e72d97', 'type': 'alarm', 'uuid': '31c4dce7-9b58-4cb2-94ef-dc07e0925bf5'} 09/11/2025 04:16:43.834 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-8da5805d-ce6f-4fbc-b2eb-c419ef085c4e', 'type': 'alarm', 'uuid': '35603741-06a3-41e6-8e94-4602b1d9ef9c'} 09/11/2025 04:16:44.026 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'oper': 'UPDATE', 'request-id': 'req-56f17f5d-7c0f-41f9-b2bb-9ffacc7f36ca', 'type': 'alarm', 'uuid': 'c072df54-74ca-459d-ae8a-19dfc9e410f0'} 09/11/2025 04:16:47.355 7fa025aa1588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:16:47.356 7fa025aa1588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:16:47.357 7fa025aa1588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:16:47.357 7fa025aa1588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:47.357 7fa025aa1588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:16:47.357 7fa025aa1588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:47.357 7fa025aa1588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:16:47.357 7fa025aa1588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:16:47.357 7fa025aa1588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:47.358 7fa025aa1588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:47.358 7fa025aa1588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:47.358 7fa025aa1588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:47.358 7fa025aa1588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:16:47.358 7fa025aa1588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:51.358 7fa0345c5198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:16:51.359 7fa0345c5198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:16:51.359 7fa0345c5198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:16:51.359 7fa0345c5198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:51.360 7fa0345c5198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:16:51.360 7fa0345c5198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:51.360 7fa0345c5198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:16:51.360 7fa0345c5198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:16:51.360 7fa0345c5198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:51.360 7fa0345c5198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:51.360 7fa0345c5198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:51.360 7fa0345c5198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:51.360 7fa0345c5198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:16:51.361 7fa0345c5198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:55.360 7fa0345c5198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:16:55.362 7fa0345c5198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:16:55.362 7fa0345c5198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:16:55.362 7fa0345c5198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:55.362 7fa0345c5198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:16:55.362 7fa0345c5198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:55.362 7fa0345c5198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:16:55.363 7fa0345c5198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:16:55.363 7fa0345c5198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:55.363 7fa0345c5198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:55.363 7fa0345c5198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:55.363 7fa0345c5198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:55.363 7fa0345c5198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:16:55.363 7fa0345c5198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:57.321 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'oper': 'UPDATE', 'request-id': 'req-35f5348f-0702-4e71-9e2f-00c3f6fbb482', 'type': 'alarm', 'uuid': 'a09334f9-32a2-497e-821d-ebb2d2e90012'} 09/11/2025 04:16:57.487 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'oper': 'UPDATE', 'request-id': 'req-6890a861-cd0c-4b4f-bcbe-08669a5eb3b9', 'type': 'alarm', 'uuid': '80c7444e-03b6-4591-b253-0de8be5b001a'} 09/11/2025 04:16:57.655 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'oper': 'UPDATE', 'request-id': 'req-c807a46e-afe1-4ac5-b9f2-d10d6bfa5b6d', 'type': 'alarm', 'uuid': '417738c3-df59-478d-bc5a-277e38b4ce0f'} 09/11/2025 04:16:57.831 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-8ad10710-8df4-473e-af65-6310ef3e6489', 'type': 'alarm', 'uuid': '95700056-2264-4db3-860a-776ac9d78bed'} 09/11/2025 04:16:57.999 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'oper': 'UPDATE', 'request-id': 'req-f5d58b6b-8629-49cc-832d-efbc93c5af2b', 'type': 'alarm', 'uuid': 'cd677829-e129-4993-aa25-30f7950c98ff'} 09/11/2025 04:16:58.165 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'oper': 'UPDATE', 'request-id': 'req-5d2832e8-3e53-4154-908e-506af2dcbc5c', 'type': 'alarm', 'uuid': 'c2334de9-48d2-49c0-8f0f-049ee80647f8'} 09/11/2025 04:16:58.337 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'oper': 'UPDATE', 'request-id': 'req-66008662-7854-44fd-92a8-c485fd1e1c43', 'type': 'alarm', 'uuid': '54bf60ee-1ff2-41ce-8914-2b057e553962'} 09/11/2025 04:16:58.502 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'oper': 'UPDATE', 'request-id': 'req-8c0c85d7-d6b0-4f7a-ba82-b2b833b61fb3', 'type': 'alarm', 'uuid': '5532ff33-efb2-4a57-9471-44761388d805'} 09/11/2025 04:16:58.673 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'oper': 'UPDATE', 'request-id': 'req-4880c172-fe06-41fd-9262-16fac2721d0a', 'type': 'alarm', 'uuid': '00bf9dce-7f0a-430c-90f9-d63d6403f71f'} 09/11/2025 04:16:58.860 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-ed5c600b-39a8-4273-974c-77f567c8ffca', 'type': 'alarm', 'uuid': 'd3578e2c-1b56-4a7d-bd58-2808f93a68f3'} 09/11/2025 04:16:59.048 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'oper': 'UPDATE', 'request-id': 'req-4acf675a-d16e-4457-b3fe-5467d59c832c', 'type': 'alarm', 'uuid': 'f1310add-cf3b-4e79-8792-7d96785f9430'} 09/11/2025 04:16:59.246 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-fa989c58-f5f3-452d-bda5-3cc564925bb6', 'type': 'alarm', 'uuid': '8383487f-ea4e-4c46-8434-94aa2ad5cb27'} 09/11/2025 04:16:59.363 7fa025a9ecc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:16:59.365 7fa025a9ecc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:16:59.365 7fa025a9ecc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:16:59.365 7fa025a9ecc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:59.365 7fa025a9ecc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:16:59.365 7fa025a9ecc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:16:59.365 7fa025a9ecc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:16:59.366 7fa025a9ecc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:16:59.366 7fa025a9ecc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:59.366 7fa025a9ecc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:16:59.366 7fa025a9ecc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:59.366 7fa025a9ecc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:59.366 7fa025a9ecc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:16:59.366 7fa025a9ecc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:16:59.420 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-c77d02e0-0ad5-4e9a-8711-5428b4797045', 'type': 'alarm', 'uuid': '422ba709-424c-4493-96c6-7110ea69ca26'} 09/11/2025 04:16:59.621 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'oper': 'UPDATE', 'request-id': 'req-31f26703-b6f6-4dff-a42c-375864931050', 'type': 'alarm', 'uuid': '0951bd37-88a0-4d5d-b3d7-fe8b8abc6c60'} 09/11/2025 04:16:59.794 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'oper': 'UPDATE', 'request-id': 'req-926b9c64-5cbe-44c5-8de1-e02d4099e4f0', 'type': 'alarm', 'uuid': 'c10e5fe7-2055-43bb-904c-a7d90fdef0a4'} 09/11/2025 04:16:59.957 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'oper': 'UPDATE', 'request-id': 'req-688820b9-d2b1-44c4-b11f-4d45a573f8cd', 'type': 'alarm', 'uuid': 'f0b395b3-1a44-44ce-9556-243055a19004'} 09/11/2025 04:17:00.129 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'oper': 'UPDATE', 'request-id': 'req-2de45689-fcbd-4d89-a058-f56b6786aac2', 'type': 'alarm', 'uuid': '64de08c6-8077-44a3-9c58-05626ee2757b'} 09/11/2025 04:17:00.308 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'oper': 'UPDATE', 'request-id': 'req-786cd116-2624-4202-9e43-efbe3274e365', 'type': 'alarm', 'uuid': '0ed2d47c-1c44-4b38-aa93-4858cfb9562b'} 09/11/2025 04:17:00.486 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'oper': 'UPDATE', 'request-id': 'req-1f399a62-c763-417b-9dc9-d02b2f610133', 'type': 'alarm', 'uuid': '71e0055b-e231-496c-9b7d-4c1bb426989e'} 09/11/2025 04:17:00.649 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'oper': 'UPDATE', 'request-id': 'req-2a8c326a-deca-43d2-acdb-822d069a61f3', 'type': 'alarm', 'uuid': '33adf7c9-f7b8-4894-986d-757783f670b1'} 09/11/2025 04:17:00.829 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-dd3e8b83-e950-4e32-b6ec-13254c7487dd', 'type': 'alarm', 'uuid': '31c4dce7-9b58-4cb2-94ef-dc07e0925bf5'} 09/11/2025 04:17:01.002 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'oper': 'UPDATE', 'request-id': 'req-16cef622-158a-4349-9fb3-3e4c0fff7e97', 'type': 'alarm', 'uuid': '35603741-06a3-41e6-8e94-4602b1d9ef9c'} 09/11/2025 04:17:01.192 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'oper': 'UPDATE', 'request-id': 'req-770932e0-5af4-4417-8eaa-b8d69f9d37d2', 'type': 'alarm', 'uuid': 'c072df54-74ca-459d-ae8a-19dfc9e410f0'} 09/11/2025 04:17:02.355 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-e6436433-bce4-4c7e-8a05-217b9502e1da', 'type': 'global_vrouter_config', 'uuid': 'b6b25520-beae-4e43-885e-8e9d517df70c'} 09/11/2025 04:17:03.366 7fa0269c1780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:03.369 7fa0269c1780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:03.370 7fa0269c1780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:03.371 7fa0269c1780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:03.373 7fa0269c1780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:03.373 7fa0269c1780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:03.374 7fa0269c1780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:03.378 7fa0269c1780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:03.378 7fa0269c1780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:03.380 7fa0269c1780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:03.380 7fa0269c1780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:03.380 7fa0269c1780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:03.381 7fa0269c1780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:03.381 7fa0269c1780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:07.369 7fa025add8d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:07.370 7fa025add8d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:07.370 7fa025add8d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:07.371 7fa025add8d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:07.371 7fa025add8d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:07.371 7fa025add8d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:07.371 7fa025add8d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:07.371 7fa025add8d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:07.371 7fa025add8d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:07.371 7fa025add8d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:07.371 7fa025add8d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:07.371 7fa025add8d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:07.372 7fa025add8d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:07.372 7fa025add8d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:07.821 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-ae630774-13b7-4b03-aad8-012ae49c3a2a', 'type': 'global_vrouter_config', 'uuid': 'b6b25520-beae-4e43-885e-8e9d517df70c'} 09/11/2025 04:17:10.868 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'id_perms': {'created': '2025-09-11T04:17:10.861383', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:10.861383', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10544535192036714357, 'uuid_mslong': 11003617109678639790}}, 'parent_type': 'security-group', 'parent_uuid': 'dbc879fa-1587-4abd-88b1-2ad35a2ac8b6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '98b4b376-2ace-4aae-9255-b6e3caf42775'}, 'oper': 'CREATE', 'request-id': 'req-1150dd39-e17c-4b33-8af4-8aec15a1caed', 'type': 'access_control_list', 'uuid': '98b4b376-2ace-4aae-9255-b6e3caf42775'} 09/11/2025 04:17:10.892 7fa025a94390 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:egress-access-control-list url = http://10.0.0.22:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-ansible-os-4080-1:/usr/bin/contrail-schema remote_ip = 10.0.0.254 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "__no_rule__", "egress-access-control-list"], "parent_type": "security-group", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "egress-access-control-list"}} domain = default-domain project = admin user = admin >> 09/11/2025 04:17:10.922 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'id_perms': {'created': '2025-09-11T04:17:10.913299', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:10.913299', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10666969232094724474, 'uuid_mslong': 6702487493598462486}}, 'parent_type': 'security-group', 'parent_uuid': 'dbc879fa-1587-4abd-88b1-2ad35a2ac8b6', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5d0404dd-376f-4216-9408-b0002507017a'}, 'oper': 'CREATE', 'request-id': 'req-2eb2dcf4-de21-4b14-a2db-a39a24f01b52', 'type': 'access_control_list', 'uuid': '5d0404dd-376f-4216-9408-b0002507017a'} 09/11/2025 04:17:10.992 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['target:64512:8000000'], 'obj_dict': {'display_name': 'target:64512:8000000', 'fq_name': ['target:64512:8000000'], 'id_perms': {'created': '2025-09-11T04:17:10.984662', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:10.984662', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9718311332703100866, 'uuid_mslong': 13557389008828058987}}, 'perms2': {'global_access': 0, 'owner': '1c7bc00d913741dc91182b098025db9c', 'owner_access': 7, 'share': []}, 'uuid': 'bc258579-30fb-4d6b-86de-60aad26bdfc2'}, 'oper': 'CREATE', 'request-id': 'req-1bc71515-94ef-490f-83f5-eea0868c2347', 'type': 'route_target', 'uuid': 'bc258579-30fb-4d6b-86de-60aad26bdfc2'} 09/11/2025 04:17:11.015 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'oper': 'UPDATE', 'request-id': 'req-568b9cb3-b440-434d-8aef-cadab7afbbf4', 'type': 'routing_instance', 'uuid': '2e1fc63d-6d97-43e5-861a-2c3093d27091'} 09/11/2025 04:17:11.037 7fa025a94390 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000001 url = http://10.0.0.22:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-ansible-os-4080-1:/usr/bin/contrail-schema remote_ip = 10.0.0.254 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000001"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "1c7bc00d913741dc91182b098025db9c", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000001"}} domain = default-domain project = admin user = admin >> 09/11/2025 04:17:11.053 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['target:64512:8000001'], 'obj_dict': {'display_name': 'target:64512:8000001', 'fq_name': ['target:64512:8000001'], 'id_perms': {'created': '2025-09-11T04:17:11.045253', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:11.045253', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13319250192862489228, 'uuid_mslong': 9764760161470663995}}, 'perms2': {'global_access': 0, 'owner': '1c7bc00d913741dc91182b098025db9c', 'owner_access': 7, 'share': []}, 'uuid': '878365a1-8558-4d3b-b8d7-7b7b641f2a8c'}, 'oper': 'CREATE', 'request-id': 'req-cf808f1d-d713-4cf6-b9f8-1e06e8df40c3', 'type': 'route_target', 'uuid': '878365a1-8558-4d3b-b8d7-7b7b641f2a8c'} 09/11/2025 04:17:11.080 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'oper': 'UPDATE', 'request-id': 'req-3278e168-094e-4f41-836f-4a2f80ffb743', 'type': 'routing_instance', 'uuid': 'a79654b7-d507-44f7-bbb7-2b6bf4c79ce3'} 09/11/2025 04:17:11.117 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['target:64512:8000002'], 'obj_dict': {'display_name': 'target:64512:8000002', 'fq_name': ['target:64512:8000002'], 'id_perms': {'created': '2025-09-11T04:17:11.111209', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:11.111209', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12342043498439276032, 'uuid_mslong': 1483168597656095540}}, 'perms2': {'global_access': 0, 'owner': '1c7bc00d913741dc91182b098025db9c', 'owner_access': 7, 'share': []}, 'uuid': '149545fb-3d4e-4b34-ab47-bf2bbee47600'}, 'oper': 'CREATE', 'request-id': 'req-47db6bf6-9a2f-4488-b96e-7da5f4b4dd47', 'type': 'route_target', 'uuid': '149545fb-3d4e-4b34-ab47-bf2bbee47600'} 09/11/2025 04:17:11.138 7fa025a94390 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 85522317-b3f9-447d-bea5-72a8afc7db3b object_type = routing_instance identifier_name = default-domain:default-project:default-virtual-network:default-virtual-network url = http://10.0.0.22:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-4080-1:/usr/bin/contrail-schema remote_ip = 10.0.0.254 body = {"type": "routing-instance", "uuid": "85522317-b3f9-447d-bea5-72a8afc7db3b", "ref-type": "route-target", "ref-uuid": "149545fb-3d4e-4b34-ab47-bf2bbee47600", "ref-fq-name": ["target:64512:8000002"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 09/11/2025 04:17:11.161 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'oper': 'UPDATE', 'request-id': 'req-8d20350d-7580-47e8-ab88-19c83b13f8bb', 'type': 'routing_instance', 'uuid': '85522317-b3f9-447d-bea5-72a8afc7db3b'} 09/11/2025 04:17:11.221 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['target:64512:8000003'], 'obj_dict': {'display_name': 'target:64512:8000003', 'fq_name': ['target:64512:8000003'], 'id_perms': {'created': '2025-09-11T04:17:11.214124', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:11.214124', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10879539168595788064, 'uuid_mslong': 9943932710958156803}}, 'perms2': {'global_access': 0, 'owner': '1c7bc00d913741dc91182b098025db9c', 'owner_access': 7, 'share': []}, 'uuid': '89fff21d-8b0c-4c03-96fb-e33b4819c120'}, 'oper': 'CREATE', 'request-id': 'req-72ca8908-bb31-4809-9eba-9193a8514e56', 'type': 'route_target', 'uuid': '89fff21d-8b0c-4c03-96fb-e33b4819c120'} 09/11/2025 04:17:11.249 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-ebe7964e-dff4-435e-96fd-f58016cbdd63', 'type': 'routing_instance', 'uuid': '010c7f62-733c-4f41-aaaa-9712d470b6e4'} 09/11/2025 04:17:11.371 7fa026897f60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:11.373 7fa026897f60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:11.373 7fa026897f60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:11.373 7fa026897f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:11.373 7fa026897f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:11.373 7fa026897f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:11.373 7fa026897f60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:11.374 7fa026897f60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:11.374 7fa026897f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:11.374 7fa026897f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:11.374 7fa026897f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:11.374 7fa026897f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:11.374 7fa026897f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:11.374 7fa026897f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:11.381 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'deny'}, 'direction': '<>', 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000002'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}], 'dynamic': False}, 'display_name': 'ip-fabric', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'id_perms': {'created': '2025-09-11T04:17:11.323519', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:11.323519', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11541561305630520375, 'uuid_mslong': 4794838566741166221}}, 'parent_type': 'virtual-network', 'parent_uuid': '81139561-8493-4c6f-b706-f85f7b7a0461', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '428ab045-b5f3-488d-a02b-dcdb8f803437'}, 'oper': 'CREATE', 'request-id': 'req-96e67f21-7a48-467e-9b31-c02b3f3598b3', 'type': 'access_control_list', 'uuid': '428ab045-b5f3-488d-a02b-dcdb8f803437'} 09/11/2025 04:17:11.414 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-6a0e4766-9273-424f-b61e-5bcd665d7389', 'type': 'access_control_list', 'uuid': '98b4b376-2ace-4aae-9255-b6e3caf42775'} 09/11/2025 04:17:11.424 7fa025a94390 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 5d0404dd-376f-4216-9408-b0002507017a object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:egress-access-control-list url = http://10.0.0.22:8082/access-control-list/5d0404dd-376f-4216-9408-b0002507017a operation = http_put useragent = cn-jenkins-deploy-platform-ansible-os-4080-1:/usr/bin/contrail-schema remote_ip = 10.0.0.254 body = {"access-control-list": {"uuid": "5d0404dd-376f-4216-9408-b0002507017a", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "access_control_list_hash": -782339169199063927}} domain = default-domain project = admin user = admin >> 09/11/2025 04:17:11.438 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-2b8be160-93d6-43a8-9ac3-118228758742', 'type': 'access_control_list', 'uuid': '5d0404dd-376f-4216-9408-b0002507017a'} 09/11/2025 04:17:13.014 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-3fab18a7-25da-48d8-921c-0fd6344da44f', 'type': 'global_vrouter_config', 'uuid': 'b6b25520-beae-4e43-885e-8e9d517df70c'} 09/11/2025 04:17:13.078 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-3'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.14', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.14', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-3', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-3'], 'id_perms': {'created': '2025-09-11T04:17:13.065074', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:13.065074', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10950853773355723412, 'uuid_mslong': 15674887581649094083}}, 'parent_type': 'routing-instance', 'parent_uuid': 'ce762e2f-e373-44d9-bd23-ef7708ab3833', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd9886334-d660-45c3-97f9-3f7c9c1d2a94'}, 'oper': 'CREATE', 'request-id': 'req-afbd7e12-b185-49f5-886c-87dd86516e9c', 'type': 'bgp_router', 'uuid': 'd9886334-d660-45c3-97f9-3f7c9c1d2a94'} 09/11/2025 04:17:13.078 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/11/2025 04:17:13.079 7fa025a7bcc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.079 7fa025a7bcc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.14", "address": "10.20.0.14", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-4080-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 15674887581649094083, "uuid_lslong": 10950853773355723412}, "enable": true, "created": "2025-09-11T04:17:13.065074", "last_modified": "2025-09-11T04:17:13.065074", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "ce762e2f-e373-44d9-bd23-ef7708ab3833"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "d9886334-d660-45c3-97f9-3f7c9c1d2a94" } deleted = False >> 09/11/2025 04:17:13.079 7fa025a7bcc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.110 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-12ea4cf1-1982-40e9-8227-2fd37ceac1c6', 'type': 'global_vrouter_config', 'uuid': 'b6b25520-beae-4e43-885e-8e9d517df70c'} 09/11/2025 04:17:13.126 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-1'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.17', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.17', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-1', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-1'], 'id_perms': {'created': '2025-09-11T04:17:13.110404', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:13.110404', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13227956060664994284, 'uuid_mslong': 12557355306522658542}}, 'parent_type': 'routing-instance', 'parent_uuid': 'ce762e2f-e373-44d9-bd23-ef7708ab3833', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ae44b01e-9ba6-46ee-b793-23f3d46999ec'}, 'oper': 'CREATE', 'request-id': 'req-cfe0e8fa-3755-4301-a4cf-7b305fcd1bc9', 'type': 'bgp_router', 'uuid': 'ae44b01e-9ba6-46ee-b793-23f3d46999ec'} 09/11/2025 04:17:13.126 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/11/2025 04:17:13.127 7fa025a648d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.127 7fa025a648d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.17", "address": "10.20.0.17", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-4080-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12557355306522658542, "uuid_lslong": 13227956060664994284}, "enable": true, "created": "2025-09-11T04:17:13.110404", "last_modified": "2025-09-11T04:17:13.110404", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "ce762e2f-e373-44d9-bd23-ef7708ab3833"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ae44b01e-9ba6-46ee-b793-23f3d46999ec" } deleted = False >> 09/11/2025 04:17:13.127 7fa025a648d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.162 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-2'], 'obj_dict': {'bgp_router_parameters': {'address': '10.20.0.254', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.20.0.254', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-2', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-2'], 'id_perms': {'created': '2025-09-11T04:17:13.149684', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:13.149684', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13535373795627972135, 'uuid_mslong': 7626155110244765493}}, 'parent_type': 'routing-instance', 'parent_uuid': 'ce762e2f-e373-44d9-bd23-ef7708ab3833', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '69d58baa-ec2b-4f35-bbd7-4ec0ee893627'}, 'oper': 'CREATE', 'request-id': 'req-50f57a94-e349-42d4-a0d0-1ed1ea7310a3', 'type': 'bgp_router', 'uuid': '69d58baa-ec2b-4f35-bbd7-4ec0ee893627'} 09/11/2025 04:17:13.163 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/11/2025 04:17:13.163 7fa025a6c978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.163 7fa025a6c978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.254", "address": "10.20.0.254", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-ansible-os-4080-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7626155110244765493, "uuid_lslong": 13535373795627972135}, "enable": true, "created": "2025-09-11T04:17:13.149684", "last_modified": "2025-09-11T04:17:13.149684", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "ce762e2f-e373-44d9-bd23-ef7708ab3833"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "69d58baa-ec2b-4f35-bbd7-4ec0ee893627" } deleted = False >> 09/11/2025 04:17:13.164 7fa025a6c978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.167 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-cd0eb5dc-62ab-4218-9e84-27c77d99ef96', 'type': 'global_vrouter_config', 'uuid': 'b6b25520-beae-4e43-885e-8e9d517df70c'} 09/11/2025 04:17:13.201 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-1'], 'oper': 'UPDATE', 'request-id': 'req-c7ff6149-09b0-4f6c-9acd-100e41a776ae', 'type': 'bgp_router', 'uuid': 'ae44b01e-9ba6-46ee-b793-23f3d46999ec'} 09/11/2025 04:17:13.223 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-3'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-c7ff6149-09b0-4f6c-9acd-100e41a776ae', 'type': 'bgp_router', 'uuid': 'd9886334-d660-45c3-97f9-3f7c9c1d2a94'} 09/11/2025 04:17:13.224 7fa028eebd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.224 7fa028eebd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.17", "address": "10.20.0.17", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "d9886334-d660-45c3-97f9-3f7c9c1d2a94"}]display_name : "cn-jenkins-deploy-platform-ansible-os-4080-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12557355306522658542, "uuid_lslong": 13227956060664994284}, "enable": true, "created": "2025-09-11T04:17:13.110404", "last_modified": "2025-09-11T04:17:13.191343", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "ce762e2f-e373-44d9-bd23-ef7708ab3833"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ae44b01e-9ba6-46ee-b793-23f3d46999ec" } deleted = False >> 09/11/2025 04:17:13.224 7fa028eebd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.235 7fa025aa1ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.235 7fa025aa1ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.14", "address": "10.20.0.14", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ae44b01e-9ba6-46ee-b793-23f3d46999ec"}]display_name : "cn-jenkins-deploy-platform-ansible-os-4080-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 15674887581649094083, "uuid_lslong": 10950853773355723412}, "enable": true, "created": "2025-09-11T04:17:13.065074", "last_modified": "2025-09-11T04:17:13.189108", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "ce762e2f-e373-44d9-bd23-ef7708ab3833"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "d9886334-d660-45c3-97f9-3f7c9c1d2a94" } deleted = False >> 09/11/2025 04:17:13.235 7fa025aa1ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.314 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-2'], 'oper': 'UPDATE', 'request-id': 'req-c769b69a-455d-415b-901c-654d46e0bb48', 'type': 'bgp_router', 'uuid': '69d58baa-ec2b-4f35-bbd7-4ec0ee893627'} 09/11/2025 04:17:13.328 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-3'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-c769b69a-455d-415b-901c-654d46e0bb48', 'type': 'bgp_router', 'uuid': 'd9886334-d660-45c3-97f9-3f7c9c1d2a94'} 09/11/2025 04:17:13.329 7fa025a65630 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.330 7fa025a65630 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.254", "address": "10.20.0.254", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "d9886334-d660-45c3-97f9-3f7c9c1d2a94"}]display_name : "cn-jenkins-deploy-platform-ansible-os-4080-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7626155110244765493, "uuid_lslong": 13535373795627972135}, "enable": true, "created": "2025-09-11T04:17:13.149684", "last_modified": "2025-09-11T04:17:13.304107", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "ce762e2f-e373-44d9-bd23-ef7708ab3833"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "69d58baa-ec2b-4f35-bbd7-4ec0ee893627" } deleted = False >> 09/11/2025 04:17:13.330 7fa025a65630 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.337 7fa025a94390 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 69d58baa-ec2b-4f35-bbd7-4ec0ee893627 object_type = bgp_router identifier_name = default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-ansible-os-4080-2 url = http://10.0.0.22:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-ansible-os-4080-1:/usr/bin/contrail-schema remote_ip = 10.0.0.254 body = {"type": "bgp-router", "uuid": "69d58baa-ec2b-4f35-bbd7-4ec0ee893627", "ref-type": "bgp-router", "ref-uuid": null, "ref-fq-name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-1"], "operation": "ADD", "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}} domain = default-domain project = admin user = admin >> 09/11/2025 04:17:13.348 7fa025a18390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.349 7fa025a18390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-3 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.14", "address": "10.20.0.14", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-2"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "69d58baa-ec2b-4f35-bbd7-4ec0ee893627"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ae44b01e-9ba6-46ee-b793-23f3d46999ec"}]display_name : "cn-jenkins-deploy-platform-ansible-os-4080-3"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-3"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 15674887581649094083, "uuid_lslong": 10950853773355723412}, "enable": true, "created": "2025-09-11T04:17:13.065074", "last_modified": "2025-09-11T04:17:13.301619", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "ce762e2f-e373-44d9-bd23-ef7708ab3833"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "d9886334-d660-45c3-97f9-3f7c9c1d2a94" } deleted = False >> 09/11/2025 04:17:13.349 7fa025a18390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.393 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-2'], 'oper': 'UPDATE', 'request-id': 'req-d7044c16-777c-4714-8fcd-9314df07aaff', 'type': 'bgp_router', 'uuid': '69d58baa-ec2b-4f35-bbd7-4ec0ee893627'} 09/11/2025 04:17:13.399 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-ansible-os-4080-1'], 'oper': 'UPDATE-IMPLICIT', 'request-id': 'req-d7044c16-777c-4714-8fcd-9314df07aaff', 'type': 'bgp_router', 'uuid': 'ae44b01e-9ba6-46ee-b793-23f3d46999ec'} 09/11/2025 04:17:13.400 7fa025a38e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.400 7fa025a38e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.254", "address": "10.20.0.254", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-1"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "ae44b01e-9ba6-46ee-b793-23f3d46999ec"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "d9886334-d660-45c3-97f9-3f7c9c1d2a94"}]display_name : "cn-jenkins-deploy-platform-ansible-os-4080-2"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-2"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7626155110244765493, "uuid_lslong": 13535373795627972135}, "enable": true, "created": "2025-09-11T04:17:13.149684", "last_modified": "2025-09-11T04:17:13.373369", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "ce762e2f-e373-44d9-bd23-ef7708ab3833"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "69d58baa-ec2b-4f35-bbd7-4ec0ee893627" } deleted = False >> 09/11/2025 04:17:13.400 7fa025a38e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.422 7fa025a1c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.422 7fa025a1c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.20.0.17", "address": "10.20.0.17", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}bgp_router_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-2"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "69d58baa-ec2b-4f35-bbd7-4ec0ee893627"}, {"to": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-3"], "attr": {"session": [{"uuid": null, "attributes": [{"bgp_router": null, "admin_down": false, "passive": false, "as_override": false, "hold_time": 0, "loop_count": 0, "local_autonomous_system": 0, "address_families": {"family": []}, "auth_data": null, "family_attributes": [], "private_as_action": null, "route_origin_override": null}]}]}, "uuid": "d9886334-d660-45c3-97f9-3f7c9c1d2a94"}]display_name : "cn-jenkins-deploy-platform-ansible-os-4080-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-ansible-os-4080-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12557355306522658542, "uuid_lslong": 13227956060664994284}, "enable": true, "created": "2025-09-11T04:17:13.110404", "last_modified": "2025-09-11T04:17:13.368104", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "ce762e2f-e373-44d9-bd23-ef7708ab3833"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ae44b01e-9ba6-46ee-b793-23f3d46999ec" } deleted = False >> 09/11/2025 04:17:13.422 7fa025a1c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:13.441 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-f2bca14f-2cfd-43bc-9948-f729c0e3a37f', 'type': 'global_system_config', 'uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb'} 09/11/2025 04:17:13.454 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-662265ce-3fdf-4280-8c7e-f94c11ba1c21', 'type': 'global_system_config', 'uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb'} 09/11/2025 04:17:13.506 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-427484c6-8284-4f5e-86b0-5f87cd9dc220', 'type': 'global_system_config', 'uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb'} 09/11/2025 04:17:14.483 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-b2522a8a-d8a5-4228-8000-92edc4f78777', 'type': 'global_system_config', 'uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb'} 09/11/2025 04:17:14.536 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-a01ff24e-c0f4-4c9a-ba1f-ec6c028d5c7b', 'type': 'global_system_config', 'uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb'} 09/11/2025 04:17:14.550 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-133c4bf2-8da3-4a02-afc8-2c066f8636f4', 'type': 'global_vrouter_config', 'uuid': 'b6b25520-beae-4e43-885e-8e9d517df70c'} 09/11/2025 04:17:15.359 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-588f12d9-dc0b-4e79-b75b-b1b0676d854e', 'type': 'global_system_config', 'uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb'} 09/11/2025 04:17:15.373 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:15.375 7fa025a180f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:15.375 7fa025a180f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:15.375 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:15.375 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:15.375 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:15.375 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:15.376 7fa025a180f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:15.376 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:15.376 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:15.376 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:15.376 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:15.376 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:15.376 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:19.376 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:19.377 7fa025a180f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:19.377 7fa025a180f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:19.378 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:19.378 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:19.378 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:19.378 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:19.378 7fa025a180f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:19.378 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:19.378 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:19.379 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:19.379 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:19.379 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:19.379 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:23.379 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:23.380 7fa025a180f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:23.380 7fa025a180f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:23.380 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:23.381 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:23.381 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:23.381 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:23.382 7fa025a180f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:23.382 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:23.382 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:23.382 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:23.382 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:23.382 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:23.382 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:27.381 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:27.383 7fa025a180f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:27.383 7fa025a180f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:27.383 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:27.384 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:27.384 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:27.384 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:27.384 7fa025a180f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:27.384 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:27.384 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:27.384 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:27.384 7fa025a180f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:27.385 7fa025a180f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:27.385 7fa025a180f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:31.384 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:31.386 7fa025f3f390 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:31.386 7fa025f3f390 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:31.386 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:31.386 7fa025f3f390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:31.386 7fa025f3f390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:31.386 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:31.386 7fa025f3f390 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:31.387 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:31.387 7fa025f3f390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:31.387 7fa025f3f390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:31.387 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:31.387 7fa025f3f390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:31.387 7fa025f3f390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:35.387 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:35.389 7fa025f3f390 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:35.389 7fa025f3f390 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:35.389 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:35.389 7fa025f3f390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:35.390 7fa025f3f390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:35.390 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:35.390 7fa025f3f390 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:35.390 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:35.390 7fa025f3f390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:35.391 7fa025f3f390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:35.391 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:35.391 7fa025f3f390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:35.391 7fa025f3f390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:39.389 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:39.391 7fa025f3f390 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:39.391 7fa025f3f390 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:39.391 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:39.391 7fa025f3f390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:39.391 7fa025f3f390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:39.391 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:39.392 7fa025f3f390 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:39.392 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:39.392 7fa025f3f390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:39.392 7fa025f3f390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:39.392 7fa025f3f390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:39.392 7fa025f3f390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:39.392 7fa025f3f390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:42.497 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-2.'], 'obj_dict': {'database_node_ip_address': '10.0.0.22', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-2.'], 'id_perms': {'created': '2025-09-11T04:17:42.484564', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:42.484564', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12473299482601648356, 'uuid_mslong': 6840234069582499767}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '5eed64a5-29fd-4fb7-ad1a-0fcb0588f0e4'}, 'oper': 'CREATE', 'request-id': 'req-15be101a-4cb8-41fa-b100-2dd43fd1e5fb', 'type': 'database_node', 'uuid': '5eed64a5-29fd-4fb7-ad1a-0fcb0588f0e4'} 09/11/2025 04:17:42.497 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/11/2025 04:17:42.498 7fa0269c1630 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:42.498 7fa0269c1630 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. elements = { database_node_ip_address : "10.0.0.22"display_name : "cn-jenkins-deploy-platform-ansible-os-4080-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-4080-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 6840234069582499767, "uuid_lslong": 12473299482601648356}, "enable": true, "created": "2025-09-11T04:17:42.484564", "last_modified": "2025-09-11T04:17:42.484564", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "197ee536-4af6-414f-b479-ed8e4f1ff5bb"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "5eed64a5-29fd-4fb7-ad1a-0fcb0588f0e4" } deleted = False >> 09/11/2025 04:17:42.498 7fa0269c1630 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:42.512 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-3.'], 'obj_dict': {'database_node_ip_address': '10.0.0.48', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-3.'], 'id_perms': {'created': '2025-09-11T04:17:42.490086', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:42.490086', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9542536437094928100, 'uuid_mslong': 9671215830728393113}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '86370f8e-c8b3-4199-846d-e654bd43cee4'}, 'oper': 'CREATE', 'request-id': 'req-24389133-93cd-48c9-81cc-32d609ab6a35', 'type': 'database_node', 'uuid': '86370f8e-c8b3-4199-846d-e654bd43cee4'} 09/11/2025 04:17:42.512 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/11/2025 04:17:42.513 7fa025afc240 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:42.513 7fa025afc240 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-3. elements = { database_node_ip_address : "10.0.0.48"display_name : "cn-jenkins-deploy-platform-ansible-os-4080-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-4080-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 9671215830728393113, "uuid_lslong": 9542536437094928100}, "enable": true, "created": "2025-09-11T04:17:42.490086", "last_modified": "2025-09-11T04:17:42.490086", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "197ee536-4af6-414f-b479-ed8e4f1ff5bb"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "86370f8e-c8b3-4199-846d-e654bd43cee4" } deleted = False >> 09/11/2025 04:17:42.513 7fa025afc240 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:42.558 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-1.'], 'obj_dict': {'database_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-1.'], 'id_perms': {'created': '2025-09-11T04:17:42.533712', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:17:42.533712', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9678306055618151222, 'uuid_mslong': 7231834154468919717}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '645ca2d8-fba8-49a5-8650-401468a1c336'}, 'oper': 'CREATE', 'request-id': 'req-996024bf-9c15-41fc-9045-02c917db3c7b', 'type': 'database_node', 'uuid': '645ca2d8-fba8-49a5-8650-401468a1c336'} 09/11/2025 04:17:42.558 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/11/2025 04:17:42.559 7fa025a7bd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:42.559 7fa025a7bd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-1. elements = { database_node_ip_address : "10.0.0.254"display_name : "cn-jenkins-deploy-platform-ansible-os-4080-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-4080-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 7231834154468919717, "uuid_lslong": 9678306055618151222}, "enable": true, "created": "2025-09-11T04:17:42.533712", "last_modified": "2025-09-11T04:17:42.533712", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "197ee536-4af6-414f-b479-ed8e4f1ff5bb"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "645ca2d8-fba8-49a5-8650-401468a1c336" } deleted = False >> 09/11/2025 04:17:42.559 7fa025a7bd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:43.392 7fa025a7bd68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:43.393 7fa025a7bd68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:43.394 7fa025a7bd68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:43.394 7fa025a7bd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:43.394 7fa025a7bd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:43.394 7fa025a7bd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:43.394 7fa025a7bd68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:43.395 7fa025a7bd68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:43.395 7fa025a7bd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:43.395 7fa025a7bd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:43.395 7fa025a7bd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:43.395 7fa025a7bd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:43.395 7fa025a7bd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:43.395 7fa025a7bd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:47.395 7fa025a7bd68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:47.396 7fa025a7bd68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:47.396 7fa025a7bd68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:47.396 7fa025a7bd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:47.397 7fa025a7bd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:47.397 7fa025a7bd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:47.397 7fa025a7bd68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:47.397 7fa025a7bd68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:47.397 7fa025a7bd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:47.397 7fa025a7bd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:47.398 7fa025a7bd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:47.398 7fa025a7bd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:47.398 7fa025a7bd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:47.398 7fa025a7bd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:51.397 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:51.398 7fa035c540f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:51.398 7fa035c540f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:51.398 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:51.398 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:51.399 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:51.399 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:51.399 7fa035c540f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:51.399 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:51.399 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:51.399 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:51.399 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:51.399 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:51.399 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:55.400 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:55.401 7fa035c540f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:55.401 7fa035c540f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:55.401 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:55.402 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:55.402 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:55.402 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:55.402 7fa035c540f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:55.402 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:55.402 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:55.402 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:55.402 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:55.403 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:55.403 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:59.403 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:17:59.404 7fa035c540f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:17:59.404 7fa035c540f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:17:59.404 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:59.404 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:17:59.404 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:17:59.404 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:17:59.405 7fa035c540f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:17:59.405 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:59.405 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:17:59.405 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:59.405 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:17:59.406 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.48:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:17:59.406 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:03.405 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:18:03.407 7fa035c540f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:18:03.407 7fa035c540f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:18:03.407 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:18:03.407 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:18:03.407 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:18:03.407 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:18:03.408 7fa035c540f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:18:03.408 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:03.408 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.254:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:18:03.408 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:03.408 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:03.408 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.254:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:18:03.408 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:07.409 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:18:07.411 7fa035c540f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 09/11/2025 04:18:07.411 7fa035c540f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:18:07.411 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:18:07.411 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:18:07.412 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:18:07.412 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 09/11/2025 04:18:07.412 7fa035c540f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 09/11/2025 04:18:07.412 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:07.412 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.22:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 09/11/2025 04:18:07.412 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:07.412 7fa035c540f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:07.413 7fa035c540f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2.:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1757564191348830 collector_name = collector_ip = 10.0.0.22:8086 collector_list = [ 10.0.0.48:8086, 10.0.0.254:8086, 10.0.0.22:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >> 09/11/2025 04:18:07.413 7fa035c540f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:10.039 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-3.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.48', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-3.'], 'id_perms': {'created': '2025-09-11T04:18:10.032351', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:18:10.032351', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13110532886506227467, 'uuid_mslong': 10416003540305922893}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '908d1408-9725-4b4d-b5f1-f8329c47c70b'}, 'oper': 'CREATE', 'request-id': 'req-f3f1b796-0cb5-48e2-894c-b30b94e1991a', 'type': 'analytics_node', 'uuid': '908d1408-9725-4b4d-b5f1-f8329c47c70b'} 09/11/2025 04:18:10.040 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/11/2025 04:18:10.041 7fa025a808d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:10.041 7fa025a808d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-3. elements = { analytics_node_ip_address : "10.0.0.48"display_name : "cn-jenkins-deploy-platform-ansible-os-4080-3."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-4080-3."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 10416003540305922893, "uuid_lslong": 13110532886506227467}, "enable": true, "created": "2025-09-11T04:18:10.032351", "last_modified": "2025-09-11T04:18:10.032351", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "197ee536-4af6-414f-b479-ed8e4f1ff5bb"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "908d1408-9725-4b4d-b5f1-f8329c47c70b" } deleted = False >> 09/11/2025 04:18:10.041 7fa025a808d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:10.069 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-2.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.22', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-2.'], 'id_perms': {'created': '2025-09-11T04:18:10.062496', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:18:10.062496', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11092590718447150403, 'uuid_mslong': 12330039933738634339}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ab1d19fe-0cdf-4c63-99f0-cc7cc3d66543'}, 'oper': 'CREATE', 'request-id': 'req-9767410a-2007-4774-9bf3-d38c886db553', 'type': 'analytics_node', 'uuid': 'ab1d19fe-0cdf-4c63-99f0-cc7cc3d66543'} 09/11/2025 04:18:10.070 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/11/2025 04:18:10.071 7fa025a80978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:10.071 7fa025a80978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. elements = { analytics_node_ip_address : "10.0.0.22"display_name : "cn-jenkins-deploy-platform-ansible-os-4080-2."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-4080-2."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 12330039933738634339, "uuid_lslong": 11092590718447150403}, "enable": true, "created": "2025-09-11T04:18:10.062496", "last_modified": "2025-09-11T04:18:10.062496", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "197ee536-4af6-414f-b479-ed8e4f1ff5bb"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "ab1d19fe-0cdf-4c63-99f0-cc7cc3d66543" } deleted = False >> 09/11/2025 04:18:10.071 7fa025a80978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:10.179 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-1.'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-1.'], 'id_perms': {'created': '2025-09-11T04:18:10.167292', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:18:10.167292', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10320892895518780064, 'uuid_mslong': 14353685761545292147}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c7328926-aa7d-4973-8f3b-2d67f23feaa0'}, 'oper': 'CREATE', 'request-id': 'req-55623a93-8c36-4fa5-9b3e-ae49d83d3a2e', 'type': 'analytics_node', 'uuid': 'c7328926-aa7d-4973-8f3b-2d67f23feaa0'} 09/11/2025 04:18:10.180 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/11/2025 04:18:10.180 7fa025a80588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:10.181 7fa025a80588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-1. elements = { analytics_node_ip_address : "10.0.0.254"display_name : "cn-jenkins-deploy-platform-ansible-os-4080-1."fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-ansible-os-4080-1."]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "heat_stack_owner", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14353685761545292147, "uuid_lslong": 10320892895518780064}, "enable": true, "created": "2025-09-11T04:18:10.167292", "last_modified": "2025-09-11T04:18:10.167292", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "197ee536-4af6-414f-b479-ed8e4f1ff5bb"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "c7328926-aa7d-4973-8f3b-2d67f23feaa0" } deleted = False >> 09/11/2025 04:18:10.181 7fa025a80588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 09/11/2025 04:18:11.412 7fa025a80588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 09/11/2025 04:18:11.413 7fa025a80588 [contrail-api] [INFO]: Session Event: TCP Connected 09/11/2025 04:18:11.414 7fa025a80588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 09/11/2025 04:18:11.414 7fa025a80588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:18:11.414 7fa025a80588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-ansible-os-4080-2. process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.254:2181, 10.0.0.22:2181, 10.0.0.48:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.48:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.254:9041, 10.0.0.22:9041, 10.0.0.48:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.254:5673, 10.0.0.22:5673, 10.0.0.48:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://10.0.0.254:5000/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 09/11/2025 04:18:11.414 7fa025a80588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 09/11/2025 04:18:11.414 7fa025a80588 [contrail-api] [INFO]: Processing event[EvTcpConnected] in state[Connect] 09/11/2025 04:18:11.414 7fa025a80588 [contrail-api] [DEBUG]: Send sandesh control message. uve type count # 19 09/11/2025 04:18:11.415 7fa025a80588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 09/11/2025 04:18:11.415 7fa025a80588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/11/2025 04:18:11.415 7fa025a80588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 09/11/2025 04:18:11.416 7fa025a8f978 [contrail-api] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/11/2025 04:18:11.417 7fa025a8f978 [contrail-api] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/11/2025 04:18:18.709 7fa025a8fac8 [contrail-api] [DEBUG]: Extracted sandesh length: 0000001016 09/11/2025 04:18:18.714 7fa025a8fac8 [contrail-api] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 09/11/2025 04:18:18.715 7fa026999438 [contrail-api] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 09/11/2025 04:18:18.715 7fa026999438 [contrail-api] [DEBUG]: Number of uve types in sandesh control message is 0 09/11/2025 04:18:18.715 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:ModuleCpuStateTrace] map 09/11/2025 04:18:18.716 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/11/2025 04:18:18.716 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/11/2025 04:18:18.716 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 09/11/2025 04:18:18.716 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectProjectTable:ContrailConfigTrace] map 09/11/2025 04:18:18.716 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/11/2025 04:18:18.716 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/11/2025 04:18:18.716 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/11/2025 04:18:18.717 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/11/2025 04:18:18.717 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/11/2025 04:18:18.717 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/11/2025 04:18:18.717 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/11/2025 04:18:18.717 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/11/2025 04:18:18.718 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/11/2025 04:18:18.718 7fa026999438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/11/2025 04:18:18.718 7fa026999438 [contrail-api] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 09/11/2025 04:18:18.719 7fa02b1ef6d8 [contrail-api] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/11/2025 04:18:18.723 7fa02b1ef6d8 [contrail-api] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/11/2025 04:18:18.724 7fa02d6d8240 [contrail-api] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/11/2025 04:18:18.727 7fa02d6d8240 [contrail-api] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/11/2025 04:18:24.747 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-1.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-1.'], 'id_perms': {'created': '2025-09-11T04:18:24.707512', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:18:24.707512', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10886084254431401214, 'uuid_mslong': 17330260339121471553}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f08173f5-7260-4041-9713-23f3d45b54fe'}, 'oper': 'CREATE', 'request-id': 'req-6820165f-126f-4f0b-acf8-6b3e60bbe16e', 'type': 'analytics_alarm_node', 'uuid': 'f08173f5-7260-4041-9713-23f3d45b54fe'} 09/11/2025 04:18:24.748 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/11/2025 04:18:25.197 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-2.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.22', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-2.'], 'id_perms': {'created': '2025-09-11T04:18:25.178613', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:18:25.178613', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10102359466033092250, 'uuid_mslong': 16787629223241666188}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e8f9a3d5-7849-4e8c-8c32-ca68e635169a'}, 'oper': 'CREATE', 'request-id': 'req-795b6ef7-8ed0-4c57-9637-7fb2f0526fe8', 'type': 'analytics_alarm_node', 'uuid': 'e8f9a3d5-7849-4e8c-8c32-ca68e635169a'} 09/11/2025 04:18:25.197 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/11/2025 04:18:25.241 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-3.'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.48', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-3.'], 'id_perms': {'created': '2025-09-11T04:18:25.226840', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:18:25.226840', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13473968969568835711, 'uuid_mslong': 17457101299669617906}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f2441523-eed1-4cf2-bafd-2763c1c9587f'}, 'oper': 'CREATE', 'request-id': 'req-ca062d57-5af5-4949-aabd-9d24207be338', 'type': 'analytics_alarm_node', 'uuid': 'f2441523-eed1-4cf2-bafd-2763c1c9587f'} 09/11/2025 04:18:25.241 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/11/2025 04:18:29.267 7fa025a8fac8 [contrail-api] [DEBUG]: Extracted sandesh length: 0000001715 09/11/2025 04:18:29.268 7fa025a8fac8 [contrail-api] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 09/11/2025 04:18:29.268 7fa026a10588 [contrail-api] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[Established] 09/11/2025 04:18:29.268 7fa026a10588 [contrail-api] [DEBUG]: Number of uve types in sandesh control message is 5 09/11/2025 04:18:29.269 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:ModuleCpuStateTrace] map 09/11/2025 04:18:29.269 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/11/2025 04:18:29.269 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/11/2025 04:18:29.269 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 09/11/2025 04:18:29.269 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectProjectTable:ContrailConfigTrace] map 09/11/2025 04:18:29.269 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/11/2025 04:18:29.269 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/11/2025 04:18:29.269 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectBgpRouter:ContrailConfigTrace] map 09/11/2025 04:18:29.270 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/11/2025 04:18:29.270 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/11/2025 04:18:29.270 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 09/11/2025 04:18:29.270 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/11/2025 04:18:29.270 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/11/2025 04:18:29.270 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 09/11/2025 04:18:29.270 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/11/2025 04:18:29.270 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/11/2025 04:18:29.270 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 09/11/2025 04:18:29.270 7fa026a10588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 09/11/2025 04:18:29.271 7fa025a51828 [contrail-api] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/11/2025 04:18:29.280 7fa025a51828 [contrail-api] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/11/2025 04:18:29.281 7fa025a51438 [contrail-api] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 09/11/2025 04:18:29.281 7fa025a51438 [contrail-api] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 09/11/2025 04:18:34.575 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-3.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.48', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-3.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-3.'], 'id_perms': {'created': '2025-09-11T04:18:34.556381', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:18:34.556381', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12859871060862440166, 'uuid_mslong': 11762908882525899753}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a33e3f4e-5c3e-4be9-b277-7098680812e6'}, 'oper': 'CREATE', 'request-id': 'req-ba2badd5-6a7d-42f4-b620-b38ae358bffc', 'type': 'analytics_snmp_node', 'uuid': 'a33e3f4e-5c3e-4be9-b277-7098680812e6'} 09/11/2025 04:18:34.576 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/11/2025 04:18:34.609 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-2.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.22', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-2.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-2.'], 'id_perms': {'created': '2025-09-11T04:18:34.569945', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:18:34.569945', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13018931947105579613, 'uuid_mslong': 13382945662263837741}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b9b9c62c-8797-442d-b4ac-89a0ee36725d'}, 'oper': 'CREATE', 'request-id': 'req-9f11c7ff-3932-4c6b-8b03-0b97a0ed2897', 'type': 'analytics_snmp_node', 'uuid': 'b9b9c62c-8797-442d-b4ac-89a0ee36725d'} 09/11/2025 04:18:34.609 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/11/2025 04:18:34.648 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-1.'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.254', 'display_name': 'cn-jenkins-deploy-platform-ansible-os-4080-1.', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-ansible-os-4080-1.'], 'id_perms': {'created': '2025-09-11T04:18:34.640219', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:18:34.640219', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12107081145061114528, 'uuid_mslong': 5224885903001928302}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '4882860b-d060-426e-a804-fe27c63282a0'}, 'oper': 'CREATE', 'request-id': 'req-3163e87a-19e2-4596-b219-ce25ac0941fc', 'type': 'analytics_snmp_node', 'uuid': '4882860b-d060-426e-a804-fe27c63282a0'} 09/11/2025 04:18:34.649 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 09/11/2025 04:20:12.936 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-6b86f379-9680-429f-81de-032adaea37b3', 'type': 'global_system_config', 'uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb'} 09/11/2025 04:20:12.990 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-b06529dd-01cc-4a90-85db-4a44fa612e7c', 'type': 'global_system_config', 'uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb'} 09/11/2025 04:20:13.093 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4080-2'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-4080-2', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4080-2'], 'id_perms': {'created': '2025-09-11T04:20:13.075833', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:20:13.075833', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12477051075703459198, 'uuid_mslong': 15851896452036903948}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'dbfd3fd6-0fef-440c-ad27-63d8dbb5cd7e', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.193'}, 'oper': 'CREATE', 'request-id': 'req-295a2d83-b65a-431f-9cac-46776648aeb7', 'type': 'virtual_router', 'uuid': 'dbfd3fd6-0fef-440c-ad27-63d8dbb5cd7e'} 09/11/2025 04:20:13.094 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectVRouter:ContrailConfigTrace] map 09/11/2025 04:20:13.146 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4080-1'], 'obj_dict': {'display_name': 'an-jenkins-deploy-platform-ansible-os-4080-1', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4080-1'], 'id_perms': {'created': '2025-09-11T04:20:13.137127', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:20:13.137127', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12488976851491600347, 'uuid_mslong': 11052219388841643631}}, 'parent_type': 'global-system-config', 'parent_uuid': '197ee536-4af6-414f-b479-ed8e4f1ff5bb', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '99615efa-1b97-4e6f-ad51-c246e93bb7db', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.20.0.25'}, 'oper': 'CREATE', 'request-id': 'req-3cd2042c-321c-4d4b-86a9-8bc268e6b080', 'type': 'virtual_router', 'uuid': '99615efa-1b97-4e6f-ad51-c246e93bb7db'} 09/11/2025 04:20:13.147 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectVRouter:ContrailConfigTrace] map 09/11/2025 04:20:13.467 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4080-2', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4080-2', 'vhost0'], 'id_perms': {'created': '2025-09-11T04:20:13.443967', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:20:13.443967', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9250108664057376590, 'uuid_mslong': 18073068845566217385}}, 'parent_type': 'virtual-router', 'parent_uuid': 'dbfd3fd6-0fef-440c-ad27-63d8dbb5cd7e', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'fad0705c-1d27-4ca9-805e-fcd21ee72f4e', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:fa:d0:70:5c:1d']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '81139561-8493-4c6f-b706-f85f7b7a0461'}]}, 'oper': 'CREATE', 'request-id': 'req-2abdf5d7-f36f-42d0-8c8d-c61a476362a5', 'type': 'virtual_machine_interface', 'uuid': 'fad0705c-1d27-4ca9-805e-fcd21ee72f4e'} 09/11/2025 04:20:13.467 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectVMITable:ContrailConfigTrace] map 09/11/2025 04:20:13.535 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4080-2', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-2abdf5d7-f36f-42d0-8c8d-c61a476362a5', 'type': 'virtual_machine_interface', 'uuid': 'fad0705c-1d27-4ca9-805e-fcd21ee72f4e'} 09/11/2025 04:20:13.585 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4080-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4080-1', 'vhost0'], 'id_perms': {'created': '2025-09-11T04:20:13.528026', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-09-11T04:20:13.528026', 'permissions': {'group': 'heat_stack_owner', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11741171555589873259, 'uuid_mslong': 5833699025282746537}}, 'parent_type': 'virtual-router', 'parent_uuid': '99615efa-1b97-4e6f-ad51-c246e93bb7db', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '50f5765a-f085-48a9-a2f1-055289dade6b', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:50:f5:76:5a:f0']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '81139561-8493-4c6f-b706-f85f7b7a0461'}]}, 'oper': 'CREATE', 'request-id': 'req-d44ceb72-2628-4eb8-8afe-1fcb8f39e9d4', 'type': 'virtual_machine_interface', 'uuid': '50f5765a-f085-48a9-a2f1-055289dade6b'} 09/11/2025 04:20:13.585 7fa025ec6e10 [contrail-api] [DEBUG]: Add uve in the [ObjectVMITable:ContrailConfigTrace] map 09/11/2025 04:20:13.655 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-global-system-config', 'an-jenkins-deploy-platform-ansible-os-4080-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-d44ceb72-2628-4eb8-8afe-1fcb8f39e9d4', 'type': 'virtual_machine_interface', 'uuid': '50f5765a-f085-48a9-a2f1-055289dade6b'} 09/11/2025 04:20:14.006 7fa025ec6e10 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiDebug: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-ffb15550-a1a0-4bb9-ac99-c4f994b16a5c', 'type': 'access_control_list', 'uuid': '428ab045-b5f3-488d-a02b-dcdb8f803437'} 09/11/2025 04:20:29.105 7fa025a3f978 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/11/2025 04:20:29.113 7fa025a3f978 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/11/2025 04:21:06.932 7fa026903e10 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/11/2025 04:21:06.936 7fa0275d8390 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 09/11/2025 04:22:22.097 7fa0259fb630 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 09/11/2025 04:22:22.102 7fa0259fb630 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map