03/31/2025 02:14:49.502 7f0b6081a448 [contrail-api] [INFO]: SANDESH: Logging: SYSLOG: [None] -> [LOG_LOCAL0] 03/31/2025 02:14:49.502 7f0b6081a448 [contrail-api] [INFO]: SANDESH: Trace: PRINT: [None] -> [False] 03/31/2025 02:14:49.502 7f0b6081a448 [contrail-api] [INFO]: SANDESH: Flow Logging: [None] -> [False] 03/31/2025 02:14:49.502 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to zookeeper on 10.0.0.17:2181 03/31/2025 02:14:49.502 7f0b6081a448 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:14:49.503 7f0b5a70beb8 [contrail-api] [ERROR]: Cannot write http_port 8084 to /tmp/contrail-api.0.http_port 03/31/2025 02:14:49.503 7f0b5a70beb8 [contrail-api] [ERROR]: Starting Introspect on HTTP Port 8084 03/31/2025 02:14:49.504 7f0b5a70f828 [contrail-api] [INFO]: Processing event[EvStart] in state[none] 03/31/2025 02:14:49.504 7f0b5a70f828 [contrail-api] [DEBUG]: Add uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/31/2025 02:14:49.505 7f0b5a70f828 [contrail-api] [INFO]: Sandesh Client: Event[EvStart] => State[none] -> State[Idle] 03/31/2025 02:14:49.505 7f0b5a70f828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:49.505 7f0b5a70f828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >> 03/31/2025 02:14:49.505 7f0b5a70f828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:49.505 7f0b5a70f828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:49.505 7f0b5a70f828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/31/2025 02:14:49.506 7f0b5a70f828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:49.506 7f0b5a70f828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:49.506 7f0b5a70f828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> 03/31/2025 02:14:49.506 7f0b5a70f828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:49.513 7f0b5a1bb0f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:49.513 7f0b5a1bb0f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector:Collector[none to Idle on EvStart] connection down >>, ] >> 03/31/2025 02:14:49.513 7f0b5a1bb0f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:49.581 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Connecting to database on ['10.0.0.17:9041'] 03/31/2025 02:14:49.582 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: VNCCassandra started with driver cql 03/31/2025 02:14:49.582 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: initializing cassandra cluster 03/31/2025 02:14:49.582 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster before create 03/31/2025 02:14:49.586 7f0b5a839048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:49.587 7f0b5a839048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 03/31/2025 02:14:49.587 7f0b5a839048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:49.590 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster created 03/31/2025 02:14:50.128 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster DB connected 03/31/2025 02:14:50.130 7f0b5a1c78d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:50.130 7f0b5a1c78d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Initializing description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[], Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 03/31/2025 02:14:50.130 7f0b5a1c78d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:50.139 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster init KS 03/31/2025 02:14:50.139 7f0b5a844780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:50.139 7f0b5a844780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Initializing description = >>, ] description = Collector:Collector[none to Idle on EvStart], Database:Cassandra[] connection down >>, ] >> 03/31/2025 02:14:50.139 7f0b5a844780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:14:50.200 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:50.316 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'config_db_uuid', created 03/31/2025 02:14:50.321 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:50.329 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:50.428 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'config_db_uuid' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 03/31/2025 02:14:50.432 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:50.439 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:50.535 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'useragent', created 03/31/2025 02:14:50.540 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:50.546 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:50.661 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: keyspace 'useragent' fixed with replication {'class': 'SimpleStrategy', 'replication_factor': None} 03/31/2025 02:14:50.670 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:50.670 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster KS are ready 03/31/2025 02:14:50.824 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: CassandraDriverCQL.__init__Cluster CF are ready 03/31/2025 02:14:50.832 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:51.067 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_uuid_table', created 03/31/2025 02:14:51.073 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:51.080 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:51.231 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_uuid_table' fixed with properties {'gc_grace_seconds': 864000} 03/31/2025 02:14:51.274 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:51.348 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:51.634 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_fq_name_table', created 03/31/2025 02:14:51.652 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:51.667 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:52.596 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_fq_name_table' fixed with properties {'gc_grace_seconds': 864000} 03/31/2025 02:14:52.604 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:52.612 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:52.875 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_shared_table', created 03/31/2025 02:14:52.917 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:52.990 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:53.149 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'obj_shared_table' fixed with properties {'gc_grace_seconds': 864000} 03/31/2025 02:14:53.155 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:53.165 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:53.396 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'useragent_keyval_table', created 03/31/2025 02:14:53.401 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:53.410 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock acquired. Performing schema change... 03/31/2025 02:14:53.505 7f0b5a7f4390 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:14:53.516 7f0b5a7f4390 [contrail-api] [INFO]: Session Event: TCP Connected 03/31/2025 02:14:53.516 7f0b5a7f4390 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:14:53.516 7f0b5a7f4390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:53.517 7f0b5a7f4390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Initializing description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired], Database:Cassandra[] connection down >>, ] >> 03/31/2025 02:14:53.517 7f0b5a7f4390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:14:53.517 7f0b5a7f4390 [contrail-api] [INFO]: Processing event[EvTcpConnected] in state[Connect] 03/31/2025 02:14:53.517 7f0b5a7f4390 [contrail-api] [DEBUG]: Send sandesh control message. uve type count # 19 03/31/2025 02:14:53.517 7f0b5a7f4390 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnected] => State[Connect] -> State[ClientInit] 03/31/2025 02:14:53.517 7f0b5a7f4390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/31/2025 02:14:53.517 7f0b5a7f4390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[ClientInit] 03/31/2025 02:14:53.518 7f0b5a80e0f0 [contrail-api] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/31/2025 02:14:53.518 7f0b5a80e0f0 [contrail-api] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/31/2025 02:14:53.523 7f0b5a7f2198 [contrail-api] [DEBUG]: Extracted sandesh length: 0000001012 03/31/2025 02:14:53.523 7f0b5a7f2198 [contrail-api] [DEBUG]: Received sandesh control message [SandeshCtrlServerToClient] 03/31/2025 02:14:53.524 7f0b5a1a2d68 [contrail-api] [INFO]: Processing event[EvSandeshCtrlMessageRecv] in state[ClientInit] 03/31/2025 02:14:53.525 7f0b5a1a2d68 [contrail-api] [DEBUG]: Number of uve types in sandesh control message is 0 03/31/2025 02:14:53.525 7f0b5a1a2d68 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:14:53.525 7f0b5a1a2d68 [contrail-api] [DEBUG]: send sync_uve in the [ObjectGeneratorInfo:SandeshModuleClientTrace] map 03/31/2025 02:14:53.525 7f0b5a1a2d68 [contrail-api] [INFO]: Sandesh Client: Event[EvSandeshCtrlMessageRecv] => State[ClientInit] -> State[Established] 03/31/2025 02:14:53.526 7f0b5a7efe10 [contrail-api] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/31/2025 02:14:53.527 7f0b5a7efe10 [contrail-api] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/31/2025 02:14:53.547 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: table 'useragent_keyval_table' fixed with properties {'gc_grace_seconds': 864000} 03/31/2025 02:14:53.552 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Zookeeper lock released. 03/31/2025 02:14:53.679 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: Initializing RabbitMQ connection, urls ['pyamqp://guest:guest@10.0.0.17:5673//'] 03/31/2025 02:14:53.692 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection down 03/31/2025 02:14:53.693 7f0b59aa8390 [contrail-api] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/31/2025 02:14:53.694 7f0b59aa8390 [contrail-api] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/31/2025 02:15:13.732 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ drainer connection ESTABLISHED 03/31/2025 02:15:13.749 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 03/31/2025 02:15:13.769 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 03/31/2025 02:15:13.774 7f0b5a7f22e8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection down 03/31/2025 02:15:13.779 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config']: Unknown id: global_system_config default-global-system-config 03/31/2025 02:15:13.794 7f0b5a7f22e8 [contrail-api] [WARNING]: __default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ publish connection ESTABLISHED 03/31/2025 02:15:13.829 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-global-vrouter-config']: Unknown id: global_vrouter_config default-global-system-config:default-global-vrouter-config 03/31/2025 02:15:13.872 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain']: Unknown id: domain default-domain 03/31/2025 02:15:14.027 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-fabric']: Unknown id: fabric default-global-system-config:default-fabric 03/31/2025 02:15:14.070 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-policy-management']: Unknown id: policy_management default-policy-management 03/31/2025 02:15:14.095 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-policy-management', 'default-application-policy-set']: Unknown id: application_policy_set default-policy-management:default-application-policy-set 03/31/2025 02:15:14.140 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric']: Unknown id: virtual_network default-domain:default-project:ip-fabric 03/31/2025 02:15:14.211 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric']: Unknown id: routing_instance default-domain:default-project:ip-fabric:ip-fabric 03/31/2025 02:15:14.263 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'ip-fabric', '__default__']: Unknown id: routing_instance default-domain:default-project:ip-fabric:__default__ 03/31/2025 02:15:14.317 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__link_local__']: Unknown id: virtual_network default-domain:default-project:__link_local__ 03/31/2025 02:15:14.410 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__link_local__', '__link_local__']: Unknown id: routing_instance default-domain:default-project:__link_local__:__link_local__ 03/31/2025 02:15:14.489 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network']: Unknown id: virtual_network default-domain:default-project:dci-network 03/31/2025 02:15:14.566 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network', 'dci-network']: Unknown id: routing_instance default-domain:default-project:dci-network:dci-network 03/31/2025 02:15:14.602 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'dci-network', '__default__']: Unknown id: routing_instance default-domain:default-project:dci-network:__default__ 03/31/2025 02:15:14.630 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'system-defined-bottle-request-size-limit']: Unknown id: alarm default-global-system-config:system-defined-bottle-request-size-limit 03/31/2025 02:15:14.660 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network']: Unknown id: routing_instance default-domain:default-project:default-virtual-network:default-virtual-network 03/31/2025 02:15:14.686 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', '__no_rule__']: Unknown id: security_group default-domain:default-project:__no_rule__ 03/31/2025 02:15:14.739 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-discovery-service-assignment']: Unknown id: discovery_service_assignment default-discovery-service-assignment 03/31/2025 02:15:14.764 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-global-qos-config']: Unknown id: global_qos_config default-global-system-config:default-global-qos-config 03/31/2025 02:15:14.812 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'service-chain-flat-ipam']: Unknown id: network_ipam default-domain:default-project:service-chain-flat-ipam 03/31/2025 02:15:14.857 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['label']: Unknown id: tag_type label 03/31/2025 02:15:14.883 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['application']: Unknown id: tag_type application 03/31/2025 02:15:14.917 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['tier']: Unknown id: tag_type tier 03/31/2025 02:15:14.948 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['deployment']: Unknown id: tag_type deployment 03/31/2025 02:15:15.021 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['site']: Unknown id: tag_type site 03/31/2025 02:15:15.041 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['neutron_fwaas']: Unknown id: tag_type neutron_fwaas 03/31/2025 02:15:15.072 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['neutron_tag']: Unknown id: tag_type neutron_tag 03/31/2025 02:15:15.112 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Executing db_resync on api-server instance 10.0.0.17_0 03/31/2025 02:15:15.113 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Contrail Version is(nightly) 03/31/2025 02:15:15.133 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: zk_sync_node data (None) 03/31/2025 02:15:15.133 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DBERESYNC: running dbe-resync 03/31/2025 02:15:15.143 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type application_policy_set len 1 03/31/2025 02:15:15.143 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for application_policy_set 03/31/2025 02:15:15.152 7f0b59acf240 [contrail-api] [DEBUG]: Add uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 03/31/2025 02:15:15.153 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for application_policy_set 03/31/2025 02:15:15.153 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_system_config len 1 03/31/2025 02:15:15.153 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_system_config 03/31/2025 02:15:15.161 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_system_config 03/31/2025 02:15:15.161 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type tag_type len 7 03/31/2025 02:15:15.162 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for tag_type 03/31/2025 02:15:15.200 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for tag_type 03/31/2025 02:15:15.200 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type namespace len 1 03/31/2025 02:15:15.200 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for namespace 03/31/2025 02:15:15.211 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for namespace 03/31/2025 02:15:15.211 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type network_ipam len 2 03/31/2025 02:15:15.211 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for network_ipam 03/31/2025 02:15:15.218 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for network_ipam 03/31/2025 02:15:15.218 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type virtual_network len 4 03/31/2025 02:15:15.219 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for virtual_network 03/31/2025 02:15:15.235 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Error in _create_default_rbac_rule: NoIdError: Unknown id: api_access_list default-global-system-config:default-api-access-list 03/31/2025 02:15:15.238 7f0b6081a448 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-global-system-config', 'default-api-access-list']: Unknown id: api_access_list default-global-system-config:default-api-access-list 03/31/2025 02:15:15.286 7f0b5a7f4a20 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 03/31/2025 02:15:15.286 7f0b5e351240 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 03/31/2025 02:15:15.287 7f0b5e617cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 03/31/2025 02:15:15.287 7f0b5d7f9f60 [contrail-api] [DEBUG]: Add uve in the [ObjectVNTable:ContrailConfigTrace] map 03/31/2025 02:15:15.288 7f0b5b61ee10 [contrail-api] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/31/2025 02:15:15.289 7f0b5b61ee10 [contrail-api] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/31/2025 02:15:15.290 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for virtual_network 03/31/2025 02:15:15.290 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type routing_instance len 6 03/31/2025 02:15:15.290 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for routing_instance 03/31/2025 02:15:15.610 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting consumer greenlet 03/31/2025 02:15:15.610 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 03/31/2025 02:15:15.610 7f0b59577828 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Starting publisher greenlet 03/31/2025 02:15:15.611 7f0b59577828 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Publisher connection 03/31/2025 02:15:15.613 7f0b6081a448 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Added consumer: amqp_request.cn-jenkins-deploy-platform-juju-os-494-1.dummy 03/31/2025 02:15:15.613 7f0b6081a448 [contrail-api] [DEBUG]: Add uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:15:15.617 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for routing_instance 03/31/2025 02:15:15.618 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type discovery_service_assignment len 1 03/31/2025 02:15:15.618 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for discovery_service_assignment 03/31/2025 02:15:15.631 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for discovery_service_assignment 03/31/2025 02:15:15.631 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type security_group len 1 03/31/2025 02:15:15.631 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for security_group 03/31/2025 02:15:15.647 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for security_group 03/31/2025 02:15:15.647 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type project len 1 03/31/2025 02:15:15.648 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for project 03/31/2025 02:15:15.653 7f0b59577828 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Publisher connection established 03/31/2025 02:15:15.653 7f0b595776d8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 03/31/2025 02:15:15.666 7f0b5d753198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Entry not found in Cassandra or ZooKeeper for ['default-domain', 'default-project', 'default-application-policy-set']: Unknown id: application_policy_set default-domain:default-project:default-application-policy-set 03/31/2025 02:15:15.667 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-juju-os-494-1.dummy'] 03/31/2025 02:15:15.677 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Ensuring Consumer connection 03/31/2025 02:15:15.692 7f0b5a802588 [contrail-api] [DEBUG]: Add uve in the [ObjectProjectTable:ContrailConfigTrace] map 03/31/2025 02:15:15.693 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for project 03/31/2025 02:15:15.693 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_qos_config len 1 03/31/2025 02:15:15.693 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_qos_config 03/31/2025 02:15:15.706 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_qos_config 03/31/2025 02:15:15.706 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type global_vrouter_config len 1 03/31/2025 02:15:15.706 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for global_vrouter_config 03/31/2025 02:15:15.709 7f0b595776d8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: KombuAmqpClient: Consumer connection established 03/31/2025 02:15:15.711 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for global_vrouter_config 03/31/2025 02:15:15.711 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type domain len 1 03/31/2025 02:15:15.711 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for domain 03/31/2025 02:15:15.723 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: Created consumers ['amqp_request.cn-jenkins-deploy-platform-juju-os-494-1.dummy'] 03/31/2025 02:15:15.730 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for domain 03/31/2025 02:15:15.730 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type alarm len 1 03/31/2025 02:15:15.730 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for alarm 03/31/2025 02:15:15.734 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for alarm 03/31/2025 02:15:15.734 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type fabric len 1 03/31/2025 02:15:15.734 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for fabric 03/31/2025 02:15:15.737 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for fabric 03/31/2025 02:15:15.737 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DB walk: obj_type policy_management len 1 03/31/2025 02:15:15.737 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Start DB Resync for policy_management 03/31/2025 02:15:15.743 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Finished DB Resync for policy_management 03/31/2025 02:15:15.747 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: DBERESYNC: Completed successfully 03/31/2025 02:15:15.756 7f0b59a974e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiInfo: Cassandra DB walk completed. 03/31/2025 02:15:15.779 7f0b59a974e0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Time elapsed in resyncing db: 0:00:00.665707 03/31/2025 02:15:15.780 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'obj_dict': {'autonomous_system': 64512, 'config_version': '1.0', 'enable_4byte_as': False, 'enable_security_policy_draft': False, 'fq_name': ['default-global-system-config'], 'ibgp_auto_mesh': True, 'id_perms': {'created': '2025-03-31T02:15:13.814912', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:13.814912', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10876990112370111713, 'uuid_mslong': 960465051042661748}}, 'igmp_enable': False, 'mac_aging_time': 300, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1'}, 'oper': 'CREATE', 'request-id': 'req-75ea7513-677c-44b8-8db5-a8ac23b366b7', 'type': 'global_system_config', 'uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1'} 03/31/2025 02:15:15.782 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'obj_dict': {'enable_security_logging': True, 'encryption_mode': 'none', 'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'id_perms': {'created': '2025-03-31T02:15:13.858101', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:13.858101', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10074867416565003530, 'uuid_mslong': 2178948087039020629}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '1e3d2dbd-97fa-4e55-8bd1-1e895383b50a'}, 'oper': 'CREATE', 'request-id': 'req-41246150-c34a-42fc-8320-a0926ef1513d', 'type': 'global_vrouter_config', 'uuid': '1e3d2dbd-97fa-4e55-8bd1-1e895383b50a'} 03/31/2025 02:15:15.783 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain'], 'obj_dict': {'fq_name': ['default-domain'], 'id_perms': {'created': '2025-03-31T02:15:13.880938', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:13.880938', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12493174131294691577, 'uuid_mslong': 12616169170328240648}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'af15a304-5fa6-4208-ad60-abae44eb60f9'}, 'oper': 'CREATE', 'request-id': 'req-050dbd59-8593-4d29-9dce-82684246110d', 'type': 'domain', 'uuid': 'af15a304-5fa6-4208-ad60-abae44eb60f9'} 03/31/2025 02:15:15.786 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-namespace'], 'obj_dict': {'_type': 'namespace', '_uuid': None, 'fq_name': ['default-domain', 'default-namespace'], 'id_perms': {'created': '2025-03-31T02:15:13.907769', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:13.907769', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13353675254835917344, 'uuid_mslong': 4039820496761865610}}, 'name': 'default-namespace', 'parent_type': 'domain', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '38105356-81d5-458a-b951-c8e4ba6b3220'}, 'oper': 'CREATE', 'request-id': 'req-63f4c7ea-8658-43a3-ae2e-d7257558c3fa', 'type': 'namespace', 'uuid': '38105356-81d5-458a-b951-c8e4ba6b3220'} 03/31/2025 02:15:15.787 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'obj_dict': {'_enable_security_policy_draft': False, '_type': 'project', '_uuid': None, 'fq_name': ['default-domain', 'default-project'], 'id_perms': {'created': '2025-03-31T02:15:13.936074', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:13.936074', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10728933233511879937, 'uuid_mslong': 9941298197842380420}}, 'name': 'default-project', 'parent_type': 'domain', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '89f6960a-28a0-4684-94e4-d3ee7a5bd501'}, 'oper': 'CREATE', 'request-id': 'req-ba612914-13f9-4c0a-8ccc-0d48f053aa8e', 'type': 'project', 'uuid': '89f6960a-28a0-4684-94e4-d3ee7a5bd501'} 03/31/2025 02:15:15.788 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network'], 'obj_dict': {'_fabric_snat': False, '_flood_unknown_unicast': False, '_igmp_enable': False, '_is_provider_network': False, '_layer2_control_word': False, '_mac_aging_time': 300, '_mac_ip_learning_enable': False, '_mac_learning_enabled': False, '_pbb_etree_enable': False, '_pbb_evpn_enable': False, '_port_security_enabled': True, '_type': 'virtual-network', '_uuid': None, '_virtual_network_category': 'tenant', 'fq_name': ['default-domain', 'default-project', 'default-virtual-network'], 'id_perms': {'created': '2025-03-31T02:15:13.976410', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:13.976410', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9295975141244024731, 'uuid_mslong': 1502601803660280479}}, 'name': 'default-virtual-network', 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '14da5061-1a08-429f-8101-f02386b6379b', 'virtual_network_network_id': 1}, 'oper': 'CREATE', 'request-id': 'req-c0f982b1-4480-493d-83cc-6a7d699adb37', 'type': 'virtual_network', 'uuid': '14da5061-1a08-429f-8101-f02386b6379b'} 03/31/2025 02:15:15.789 7f0b598ed4e0 [contrail-api] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/31/2025 02:15:15.790 7f0b598ed4e0 [contrail-api] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/31/2025 02:15:15.794 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'obj_dict': {'_type': 'network-ipam', '_uuid': None, 'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'id_perms': {'created': '2025-03-31T02:15:14.007783', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.007783', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11120249945462724132, 'uuid_mslong': 14776215186794235450}}, 'name': 'default-network-ipam', 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'cd0fa96c-b59a-463a-9a53-106864dcd624'}, 'oper': 'CREATE', 'request-id': 'req-5e7de254-dfad-4e0b-b52a-cbc769013fdc', 'type': 'network_ipam', 'uuid': 'cd0fa96c-b59a-463a-9a53-106864dcd624'} 03/31/2025 02:15:15.795 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-fabric'], 'obj_dict': {'disable_vlan_vn_uniqueness_check': False, 'fabric_enterprise_style': True, 'fq_name': ['default-global-system-config', 'default-fabric'], 'id_perms': {'created': '2025-03-31T02:15:14.057252', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.057252', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9802384796268430565, 'uuid_mslong': 1388113760245728608}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '1343921c-4347-4560-8809-1109754504e5'}, 'oper': 'CREATE', 'request-id': 'req-5bdafee2-e740-49fd-a9da-4baca74d1068', 'type': 'fabric', 'uuid': '1343921c-4347-4560-8809-1109754504e5'} 03/31/2025 02:15:15.796 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management'], 'obj_dict': {'fq_name': ['default-policy-management'], 'id_perms': {'created': '2025-03-31T02:15:14.085755', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.085755', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12630495078249874708, 'uuid_mslong': 9742357483541055158}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '8733ce83-9955-46b6-af48-885abc674514'}, 'oper': 'CREATE', 'request-id': 'req-2bed1580-e18a-46fb-89da-c9a7a6ddebfc', 'type': 'policy_management', 'uuid': '8733ce83-9955-46b6-af48-885abc674514'} 03/31/2025 02:15:15.797 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'fq_name': ['default-policy-management', 'default-application-policy-set'], 'id_perms': {'created': '2025-03-31T02:15:14.114332', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.114332', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9244352699827698891, 'uuid_mslong': 8639165207785983336}}, 'parent_type': 'policy-management', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '77e47afb-f5ff-4d68-804a-89cd43a73ccb'}, 'oper': 'CREATE', 'request-id': 'req-7816a085-e4e6-4764-ba82-b0d79631160c', 'type': 'application_policy_set', 'uuid': '77e47afb-f5ff-4d68-804a-89cd43a73ccb'} 03/31/2025 02:15:15.798 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-policy-management', 'default-application-policy-set'], 'oper': 'UPDATE', 'request-id': 'req-eb515a5c-2b87-4883-8598-ac620b7e5947', 'type': 'application_policy_set', 'uuid': '77e47afb-f5ff-4d68-804a-89cd43a73ccb'} 03/31/2025 02:15:15.805 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', 'ip-fabric'], 'id_perms': {'created': '2025-03-31T02:15:14.158439', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.158439', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12218376265494429311, 'uuid_mslong': 4442675259612088628}}, 'igmp_enable': False, 'is_provider_network': True, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': '3da78d9c-3d98-4934-a990-647a4cdbfe7f', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 2}, 'oper': 'CREATE', 'request-id': 'req-8daab90c-8668-4e11-a5d8-8af1ccf0efed', 'type': 'virtual_network', 'uuid': '3da78d9c-3d98-4934-a990-647a4cdbfe7f'} 03/31/2025 02:15:15.808 7f0b598edc18 [contrail-api] [INFO]: Sandesh Send Level [INVALID] -> [SYS_DEBUG] 03/31/2025 02:15:15.809 7f0b598edc18 [contrail-api] [INFO]: Sandesh Send Level [SYS_DEBUG] -> [INVALID] 03/31/2025 02:15:15.816 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'id_perms': {'created': '2025-03-31T02:15:14.241524', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.241524', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12639210675142952246, 'uuid_mslong': 16217167539418841254}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'e10ef3f4-720e-40a6-af67-7f24c5388536'}, 'oper': 'CREATE', 'request-id': 'req-9f364c7a-5989-4ec8-a90b-16f36ee0e673', 'type': 'routing_instance', 'uuid': 'e10ef3f4-720e-40a6-af67-7f24c5388536'} 03/31/2025 02:15:15.817 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__'], 'id_perms': {'created': '2025-03-31T02:15:14.288856', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.288856', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10344582809577725557, 'uuid_mslong': 15558282908606087176}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': False, 'uuid': 'd7ea1fdf-c27f-4008-8f8f-5741e27e1e75'}, 'oper': 'CREATE', 'request-id': 'req-6bbe161b-5e23-4edb-9d11-88129eef6815', 'type': 'routing_instance', 'uuid': 'd7ea1fdf-c27f-4008-8f8f-5741e27e1e75'} 03/31/2025 02:15:15.819 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__link_local__'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', '__link_local__'], 'id_perms': {'created': '2025-03-31T02:15:14.345883', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.345883', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10737692867643652447, 'uuid_mslong': 14147005145103353515}}, 'igmp_enable': False, 'is_provider_network': False, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': 'c4544239-b582-46ab-9503-f2c5bb25395f', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 3}, 'oper': 'CREATE', 'request-id': 'req-c19fc5a8-7715-4543-b76c-5bf278dfe56b', 'type': 'virtual_network', 'uuid': 'c4544239-b582-46ab-9503-f2c5bb25395f'} 03/31/2025 02:15:15.825 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__link_local__', '__link_local__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', '__link_local__', '__link_local__'], 'id_perms': {'created': '2025-03-31T02:15:14.458196', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.458196', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10135801312914432145, 'uuid_mslong': 9056758980686595501}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '7db0124e-e427-45ad-8ca9-9997a233e091'}, 'oper': 'CREATE', 'request-id': 'req-0b236616-8643-4633-a00a-9b132918b7e9', 'type': 'routing_instance', 'uuid': '7db0124e-e427-45ad-8ca9-9997a233e091'} 03/31/2025 02:15:15.827 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network'], 'obj_dict': {'fabric_snat': False, 'flood_unknown_unicast': False, 'fq_name': ['default-domain', 'default-project', 'dci-network'], 'id_perms': {'created': '2025-03-31T02:15:14.515609', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.515609', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12502647882110616526, 'uuid_mslong': 13981553745498295758}}, 'igmp_enable': False, 'is_provider_network': False, 'layer2_control_word': False, 'mac_aging_time': 300, 'mac_ip_learning_enable': False, 'mac_learning_enabled': False, 'parent_type': 'project', 'pbb_etree_enable': False, 'pbb_evpn_enable': False, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'port_security_enabled': True, 'uuid': 'c208750d-d875-4dce-ad82-5401c4fe43ce', 'virtual_network_category': 'tenant', 'virtual_network_network_id': 4}, 'oper': 'CREATE', 'request-id': 'req-1e226ccf-8491-4f3a-9ef0-341ca5c90fc5', 'type': 'virtual_network', 'uuid': 'c208750d-d875-4dce-ad82-5401c4fe43ce'} 03/31/2025 02:15:15.843 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'id_perms': {'created': '2025-03-31T02:15:14.591489', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.591489', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11737874044451481635, 'uuid_mslong': 14673641457660873781}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': 'cba33f29-5552-4c35-a2e5-4e40e5cee823'}, 'oper': 'CREATE', 'request-id': 'req-e086d8ba-0163-496d-affe-b586754daaa3', 'type': 'routing_instance', 'uuid': 'cba33f29-5552-4c35-a2e5-4e40e5cee823'} 03/31/2025 02:15:15.844 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'id_perms': {'created': '2025-03-31T02:15:14.621378', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.621378', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11650509553945933144, 'uuid_mslong': 13116144206712817815}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': False, 'uuid': 'b605e7a9-e7ea-4897-a1ae-ecb6573df558'}, 'oper': 'CREATE', 'request-id': 'req-881ae8e5-31db-405c-b1d0-14522265ddca', 'type': 'routing_instance', 'uuid': 'b605e7a9-e7ea-4897-a1ae-ecb6573df558'} 03/31/2025 02:15:15.845 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bottle-request-size-limit'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'UveConfigReq.err_info.*.', 'operand2': {'json_value': 'True'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'fq_name': ['default-global-system-config', 'system-defined-bottle-request-size-limit'], 'id_perms': {'created': '2025-03-31T02:15:14.653035', 'description': 'Bottle request size limit exceeded.', 'last_modified': '2025-03-31T02:15:14.653035', 'uuid': {'uuid_lslong': 10514699941232632691, 'uuid_mslong': 18370428595919932277}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'fef0df7a-2f7c-4b75-91eb-b7e324fde373', 'uve_keys': {'uve_key': ['config-node']}}, 'oper': 'CREATE', 'request-id': 'req-2ee9049f-305b-4979-9546-9f783b73aef7', 'type': 'alarm', 'uuid': 'fef0df7a-2f7c-4b75-91eb-b7e324fde373'} 03/31/2025 02:15:15.847 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'id_perms': {'created': '2025-03-31T02:15:14.678026', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.678026', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12495627877503172342, 'uuid_mslong': 11038241118007020083}}, 'parent_type': 'virtual-network', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'routing_instance_fabric_snat': False, 'routing_instance_has_pnf': False, 'routing_instance_is_default': True, 'uuid': '992fb5d0-4feb-4a33-ad69-6359940b7ef6'}, 'oper': 'CREATE', 'request-id': 'req-e99963f4-be75-405f-a880-72c502365af7', 'type': 'routing_instance', 'uuid': '992fb5d0-4feb-4a33-ad69-6359940b7ef6'} 03/31/2025 02:15:15.848 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__'], 'obj_dict': {'configured_security_group_id': 0, 'fq_name': ['default-domain', 'default-project', '__no_rule__'], 'id_perms': {'created': '2025-03-31T02:15:14.714041', 'creator': None, 'description': 'Security group with no rules', 'enable': True, 'last_modified': '2025-03-31T02:15:14.714041', 'permissions': None, 'user_visible': True, 'uuid': {'uuid_lslong': 12014817010079790094, 'uuid_mslong': 15147172450522581965}}, 'parent_type': 'project', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'security_group_entries': {'policy_rule': []}, 'security_group_id': 8000001, 'uuid': 'd2359117-30fb-4bcd-a6bd-3469d720740e'}, 'oper': 'CREATE', 'request-id': 'req-d6c992d4-11aa-48ed-aa9e-af6d2f081772', 'type': 'security_group', 'uuid': 'd2359117-30fb-4bcd-a6bd-3469d720740e'} 03/31/2025 02:15:15.851 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-discovery-service-assignment'], 'obj_dict': {'fq_name': ['default-discovery-service-assignment'], 'id_perms': {'created': '2025-03-31T02:15:14.752394', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.752394', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9236179977168292479, 'uuid_mslong': 2222236393932146930}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '1ed6f839-f345-48f2-802d-80c0fcb10a7f'}, 'oper': 'CREATE', 'request-id': 'req-bb93028e-e61a-48fd-a397-abdeb9756111', 'type': 'discovery_service_assignment', 'uuid': '1ed6f839-f345-48f2-802d-80c0fcb10a7f'} 03/31/2025 02:15:15.854 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-qos-config'], 'obj_dict': {'fq_name': ['default-global-system-config', 'default-global-qos-config'], 'id_perms': {'created': '2025-03-31T02:15:14.795224', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.795224', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11809466522532915955, 'uuid_mslong': 17031384377889014496}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ec5ba1db-0536-42e0-a3e3-a73ba6712af3'}, 'oper': 'CREATE', 'request-id': 'req-e1497afe-f4d5-4b76-8a5d-bede86b9e2c4', 'type': 'global_qos_config', 'uuid': 'ec5ba1db-0536-42e0-a3e3-a73ba6712af3'} 03/31/2025 02:15:15.857 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'service-chain-flat-ipam'], 'obj_dict': {'fq_name': ['default-domain', 'default-project', 'service-chain-flat-ipam'], 'id_perms': {'created': '2025-03-31T02:15:14.846226', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.846226', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13118574096373761402, 'uuid_mslong': 16561128200475265281}}, 'ipam_subnet_method': 'flat-subnet', 'ipam_subnets': {'subnets': [{'addr_from_start': None, 'alloc_unit': 1, 'allocation_pools': [], 'created': None, 'default_gateway': None, 'dhcp_option_list': None, 'dhcp_relay_server': [], 'dns_nameservers': [], 'dns_server_address': None, 'enable_dhcp': True, 'host_routes': None, 'last_modified': None, 'subnet': {'ip_prefix': '0.0.0.0', 'ip_prefix_len': 8}, 'subnet_name': None, 'subnet_uuid': None, 'subscriber_tag': None, 'vlan_tag': None}, {'addr_from_start': None, 'alloc_unit': 1, 'allocation_pools': [], 'created': None, 'default_gateway': None, 'dhcp_option_list': None, 'dhcp_relay_server': [], 'dns_nameservers': [], 'dns_server_address': None, 'enable_dhcp': True, 'host_routes': None, 'last_modified': None, 'subnet': {'ip_prefix': '::ffff', 'ip_prefix_len': 104}, 'subnet_name': None, 'subnet_uuid': None, 'subscriber_tag': None, 'vlan_tag': None}]}, 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e5d4f25a-8d08-4901-b60e-89a2adffc57a'}, 'oper': 'CREATE', 'request-id': 'req-718d85cd-4445-4866-ac02-5753c5c5061a', 'type': 'network_ipam', 'uuid': 'e5d4f25a-8d08-4901-b60e-89a2adffc57a'} 03/31/2025 02:15:15.863 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['label'], 'obj_dict': {'display_name': 'label', 'fq_name': ['label'], 'id_perms': {'created': '2025-03-31T02:15:14.873066', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.873066', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 11747993238677306573, 'uuid_mslong': 3837255732850933777}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0000', 'uuid': '3540abc2-24e9-4011-a309-419b670d3ccd'}, 'oper': 'CREATE', 'request-id': 'req-776da9d7-2583-486c-969d-c6baf9ca674f', 'type': 'tag_type', 'uuid': '3540abc2-24e9-4011-a309-419b670d3ccd'} 03/31/2025 02:15:15.865 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['application'], 'obj_dict': {'display_name': 'application', 'fq_name': ['application'], 'id_perms': {'created': '2025-03-31T02:15:14.900313', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.900313', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 11475502756582100306, 'uuid_mslong': 16802706573726206003}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0001', 'uuid': 'e92f349a-b012-4833-9f41-2cf511178d52'}, 'oper': 'CREATE', 'request-id': 'req-2611ed82-f618-4590-a54b-fb72baf01014', 'type': 'tag_type', 'uuid': 'e92f349a-b012-4833-9f41-2cf511178d52'} 03/31/2025 02:15:15.868 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['tier'], 'obj_dict': {'display_name': 'tier', 'fq_name': ['tier'], 'id_perms': {'created': '2025-03-31T02:15:14.940835', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.940835', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 11199833532170392926, 'uuid_mslong': 9759673435306017068}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0002', 'uuid': '87715347-cb3b-412c-9b6d-cd421c86555e'}, 'oper': 'CREATE', 'request-id': 'req-e488b685-9bc7-414d-bb69-3801e80e4d75', 'type': 'tag_type', 'uuid': '87715347-cb3b-412c-9b6d-cd421c86555e'} 03/31/2025 02:15:15.874 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['deployment'], 'obj_dict': {'display_name': 'deployment', 'fq_name': ['deployment'], 'id_perms': {'created': '2025-03-31T02:15:14.969479', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:14.969479', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 13099552326006247110, 'uuid_mslong': 8677577945819401888}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0003', 'uuid': '786cf32a-7b54-4ea0-b5ca-f56fa43e8ec6'}, 'oper': 'CREATE', 'request-id': 'req-0f7ca7a5-c81a-4b25-8221-3ac0a6c94170', 'type': 'tag_type', 'uuid': '786cf32a-7b54-4ea0-b5ca-f56fa43e8ec6'} 03/31/2025 02:15:15.886 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['site'], 'obj_dict': {'display_name': 'site', 'fq_name': ['site'], 'id_perms': {'created': '2025-03-31T02:15:15.034203', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:15.034203', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 10015647312583228470, 'uuid_mslong': 4362144643513272603}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0004', 'uuid': '3c897371-0f7b-491b-8afe-ba2a661e7c36'}, 'oper': 'CREATE', 'request-id': 'req-dcc2d309-9f4b-4713-8861-7dc2aa20af1e', 'type': 'tag_type', 'uuid': '3c897371-0f7b-491b-8afe-ba2a661e7c36'} 03/31/2025 02:15:15.892 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['neutron_fwaas'], 'obj_dict': {'display_name': 'neutron_fwaas', 'fq_name': ['neutron_fwaas'], 'id_perms': {'created': '2025-03-31T02:15:15.053285', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:15.053285', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 13429195708679988471, 'uuid_mslong': 6338248257710342152}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0005', 'uuid': '57f5fb35-7b10-4008-ba5e-16589b1224f7'}, 'oper': 'CREATE', 'request-id': 'req-dc1de1e9-101e-4125-9967-36421e7e49f6', 'type': 'tag_type', 'uuid': '57f5fb35-7b10-4008-ba5e-16589b1224f7'} 03/31/2025 02:15:15.895 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['neutron_tag'], 'obj_dict': {'display_name': 'neutron_tag', 'fq_name': ['neutron_tag'], 'id_perms': {'created': '2025-03-31T02:15:15.097665', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:15.097665', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': False, 'uuid': {'uuid_lslong': 10289935221191530809, 'uuid_mslong': 7877559320351493682}}, 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'tag_type_id': '0x0006', 'uuid': '6d52b676-fa9e-4e32-8ecd-3190b4e6d939'}, 'oper': 'CREATE', 'request-id': 'req-73d4788f-8ef2-45ce-9fa0-edba186dcb66', 'type': 'tag_type', 'uuid': '6d52b676-fa9e-4e32-8ecd-3190b4e6d939'} 03/31/2025 02:15:15.899 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-network-ipam'], 'oper': 'UPDATE', 'request-id': 'req-c9ea54c0-ff7e-4451-b286-8039d6cd4755', 'type': 'network_ipam', 'uuid': 'cd0fa96c-b59a-463a-9a53-106864dcd624'} 03/31/2025 02:15:15.905 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-api-access-list'], 'obj_dict': {'api_access_list_entries': {'rbac_rule': [{'rule_field': '', 'rule_object': 'fqname-to-id', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'id-to-fqname', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'useragent-kv', 'rule_perms': [{'role_crud': 'CRUD', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'documentation', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': '/', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': '', 'rule_object': 'watch', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}, {'rule_field': 'logical_router_gateway_external', 'rule_object': 'logical-router', 'rule_perms': [{'role_crud': 'R', 'role_name': '*'}]}]}, 'fq_name': ['default-global-system-config', 'default-api-access-list'], 'id_perms': {'created': '2025-03-31T02:15:15.280431', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:15.280431', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10880358667029119941, 'uuid_mslong': 1401070152961117319}}, 'parent_type': 'global-system-config', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '137199e1-84a6-4c87-96fe-cc8fa11d03c5'}, 'oper': 'CREATE', 'request-id': 'req-f4170da7-d6cf-4e25-a54f-a43cf3bf4ae1', 'type': 'api_access_list', 'uuid': '137199e1-84a6-4c87-96fe-cc8fa11d03c5'} 03/31/2025 02:15:15.906 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-application-policy-set'], 'obj_dict': {'all_applications': True, 'fq_name': ['default-domain', 'default-project', 'default-application-policy-set'], 'id_perms': {'created': '2025-03-31T02:15:15.686768', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:15.686768', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9901748465217741933, 'uuid_mslong': 5324390320495478456}}, 'parent_type': 'project', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '49e408c9-62a9-46b8-896a-13c47492a46d'}, 'oper': 'CREATE', 'request-id': 'req-72b30bde-2f02-4a9b-9f68-bd099562515e', 'type': 'application_policy_set', 'uuid': '49e408c9-62a9-46b8-896a-13c47492a46d'} 03/31/2025 02:15:15.907 7f0b5a7f2cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectApplicationPolicySetTable:ContrailConfigTrace] map 03/31/2025 02:15:15.907 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project'], 'oper': 'UPDATE', 'request-id': 'req-de772afd-258b-486e-96a1-45d2e470cb29', 'type': 'project', 'uuid': '89f6960a-28a0-4684-94e4-d3ee7a5bd501'} 03/31/2025 02:15:16.719 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:15:16.719 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:15:16.719 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:15:16.719 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:15:16.719 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:15:16.720 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:15:16.720 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:15:16.720 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:15:16.720 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:15:16.720 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:15:16.720 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:15:16.720 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:15:16.720 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:15:16.720 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:15:16.720 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:15:16.720 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:15:16.721 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:15:16.721 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:15:16.721 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:15:16.721 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:15:16.721 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:15:16.721 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:15:16.721 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:15:16.721 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:15:16.721 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:15:16.721 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:15:16.721 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:15:16.722 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:15:16.722 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:15:16.722 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:15:16.722 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:15:16.722 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:15:16.722 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 03/31/2025 02:15:16.722 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:15:16.722 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:15:16.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:16.860 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'] not found in DB: Unknown id: analytics_alarm_node default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 03/31/2025 02:15:16.882 7f0b595c0b70 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_alarm_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 url = https://10.0.0.17:8082/analytics-alarm-nodes operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_analytics_alarm_node.py remote_ip = 10.0.0.17 body = {"analytics-alarm-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-juju-os-494-1"], "parent_type": "global-system-config", "analytics_alarm_node_ip_address": "10.0.0.17", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-juju-os-494-1"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:16.935 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'obj_dict': {'analytics_alarm_node_ip_address': '10.0.0.17', 'display_name': 'cn-jenkins-deploy-platform-juju-os-494-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'id_perms': {'created': '2025-03-31T02:15:16.917477', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:16.917477', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10009706372630506728, 'uuid_mslong': 5917222539997036706}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '521e328c-a175-40a2-8ae9-9ee9869aece8'}, 'oper': 'CREATE', 'request-id': 'req-ad433319-1a05-43c9-9a8a-d1bd7704beb1', 'type': 'analytics_alarm_node', 'uuid': '521e328c-a175-40a2-8ae9-9ee9869aece8'} 03/31/2025 02:15:16.936 7f0b5a7f2cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsAlarmInfo:ContrailConfigTrace] map 03/31/2025 02:15:17.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:17.735 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:15:17.735 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:15:17.735 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:15:17.735 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:15:17.735 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:15:17.735 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:15:17.735 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:15:17.736 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:15:17.737 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:15:17.737 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:15:17.737 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:15:17.737 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:15:17.737 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:15:17.737 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:15:17.737 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:15:17.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:15:17.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:15:17.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:15:17.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:15:17.740 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:15:17.740 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:15:17.740 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 03/31/2025 02:15:17.740 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:15:17.740 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:15:17.792 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'opencontrail'] not found in DB: Unknown id: service_appliance_set default-global-system-config:opencontrail 03/31/2025 02:15:17.793 7f0b5712ecc0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_appliance_set identifier_name = default-global-system-config:opencontrail url = https://10.0.0.17:8082/service-appliance-sets operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.17 body = {"service-appliance-set": {"uuid": null, "fq_name": ["default-global-system-config", "opencontrail"], "parent_type": "global-system-config", "service_appliance_driver": "svc_monitor.services.loadbalancer.drivers.ha_proxy.driver.OpencontrailLoadbalancerDriver", "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 5, "share": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "display_name": "opencontrail"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:17.827 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'opencontrail'], 'obj_dict': {'display_name': 'opencontrail', 'fq_name': ['default-global-system-config', 'opencontrail'], 'id_perms': {'created': '2025-03-31T02:15:17.807967', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:17.807967', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12947249837083363433, 'uuid_mslong': 13532515441455351251}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service_appliance_driver': 'svc_monitor.services.loadbalancer.drivers.ha_proxy.driver.OpencontrailLoadbalancerDriver', 'uuid': 'bbcd2718-7d6f-45d3-b3ad-df211d968869'}, 'oper': 'CREATE', 'request-id': 'req-2141807f-56a0-45c9-a5bd-3924e8510806', 'type': 'service_appliance_set', 'uuid': 'bbcd2718-7d6f-45d3-b3ad-df211d968869'} 03/31/2025 02:15:17.986 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:15:17.987 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:15:17.987 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:15:17.987 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:15:17.987 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:15:17.987 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:15:17.987 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:15:17.987 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:15:17.987 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:15:17.987 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:15:17.987 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:15:17.988 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:15:17.988 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:15:17.988 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:15:17.988 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:15:17.988 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:15:17.989 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:15:17.989 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:15:17.990 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:15:17.990 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:15:17.992 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:15:17.992 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:15:17.993 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:15:17.993 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:15:17.993 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:15:17.993 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:15:17.994 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:15:17.994 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:15:17.994 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:15:17.994 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:15:17.994 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:15:17.995 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:15:17.995 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 03/31/2025 02:15:17.996 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:15:17.996 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:15:18.045 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'native'] not found in DB: Unknown id: service_appliance_set default-global-system-config:native 03/31/2025 02:15:18.046 7f0b598128d0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_appliance_set identifier_name = default-global-system-config:native url = https://10.0.0.17:8082/service-appliance-sets operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.17 body = {"service-appliance-set": {"uuid": null, "fq_name": ["default-global-system-config", "native"], "parent_type": "global-system-config", "service_appliance_driver": "svc_monitor.services.loadbalancer.drivers.native.driver.OpencontrailLoadbalancerDriver", "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 5, "share": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "display_name": "native"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:18.066 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'native'], 'obj_dict': {'display_name': 'native', 'fq_name': ['default-global-system-config', 'native'], 'id_perms': {'created': '2025-03-31T02:15:18.060244', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:18.060244', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12363259716651826701, 'uuid_mslong': 6286466634206757003}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 5, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'service_appliance_driver': 'svc_monitor.services.loadbalancer.drivers.native.driver.OpencontrailLoadbalancerDriver', 'uuid': '573e0418-c03d-408b-ab93-1f357cedbe0d'}, 'oper': 'CREATE', 'request-id': 'req-a863a604-5804-4012-8425-3709226f289e', 'type': 'service_appliance_set', 'uuid': '573e0418-c03d-408b-ab93-1f357cedbe0d'} 03/31/2025 02:15:18.184 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_DNS_refs in resource network-ipam: 'virtual_DNS_refs' 03/31/2025 02:15:18.184 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource network-ipam: 'tag_refs' 03/31/2025 02:15:18.184 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource network-ipam: 'virtual_network_back_refs' 03/31/2025 02:15:18.184 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_router_back_refs in resource network-ipam: 'virtual_router_back_refs' 03/31/2025 02:15:18.184 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource network-ipam: 'instance_ip_back_refs' 03/31/2025 02:15:18.374 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'analyzer-template'] not found in DB: Unknown id: service_template default-domain:analyzer-template 03/31/2025 02:15:18.375 7f0b598182e8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:analyzer-template url = https://10.0.0.17:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.17 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "analyzer-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": null, "service_type": "analyzer", "image_name": "analyzer", "service_scaling": false, "interface_type": [{"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}], "flavor": "m1.medium", "ordered_interfaces": true, "service_virtualization_type": "virtual-machine", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "e808cb49d7ac4baab66085d471fe76a9", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "analyzer-template"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:18.412 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'analyzer-template'], 'obj_dict': {'display_name': 'analyzer-template', 'fq_name': ['default-domain', 'analyzer-template'], 'id_perms': {'created': '2025-03-31T02:15:18.396707', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:18.396707', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10338298482804897856, 'uuid_mslong': 13001296932609607497}}, 'parent_type': 'domain', 'parent_uuid': 'af15a304-5fa6-4208-ad60-abae44eb60f9', 'perms2': {'global_access': 0, 'owner': 'e808cb49d7ac4baab66085d471fe76a9', 'owner_access': 7, 'share': [{'tenant': 'domain:af15a304-5fa6-4208-ad60-abae44eb60f9', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': 'm1.medium', 'image_name': 'analyzer', 'instance_data': None, 'interface_type': [{'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': None, 'service_scaling': False, 'service_type': 'analyzer', 'service_virtualization_type': 'virtual-machine', 'version': 1, 'vrouter_instance_type': None}, 'uuid': 'b46de2ad-2792-4749-8f79-03b2098bb840'}, 'oper': 'CREATE', 'request-id': 'req-d6432cf1-cc99-4740-9cb0-94b043f774d3', 'type': 'service_template', 'uuid': 'b46de2ad-2792-4749-8f79-03b2098bb840'} 03/31/2025 02:15:18.427 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'nat-template'] not found in DB: Unknown id: service_template default-domain:nat-template 03/31/2025 02:15:18.428 7f0b59401390 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:nat-template url = https://10.0.0.17:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.17 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "nat-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "firewall", "image_name": "analyzer", "service_scaling": false, "interface_type": [{"service_interface_type": "management", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "right", "shared_ip": false, "static_route_enable": false}], "flavor": "m1.medium", "ordered_interfaces": true, "service_virtualization_type": "virtual-machine", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "e808cb49d7ac4baab66085d471fe76a9", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "nat-template"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:18.462 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'nat-template'], 'obj_dict': {'display_name': 'nat-template', 'fq_name': ['default-domain', 'nat-template'], 'id_perms': {'created': '2025-03-31T02:15:18.452950', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:18.452950', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11691210451396471369, 'uuid_mslong': 8336293677799588570}}, 'parent_type': 'domain', 'parent_uuid': 'af15a304-5fa6-4208-ad60-abae44eb60f9', 'perms2': {'global_access': 0, 'owner': 'e808cb49d7ac4baab66085d471fe76a9', 'owner_access': 7, 'share': [{'tenant': 'domain:af15a304-5fa6-4208-ad60-abae44eb60f9', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': 'm1.medium', 'image_name': 'analyzer', 'instance_data': None, 'interface_type': [{'service_interface_type': 'management', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'right', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': False, 'service_type': 'firewall', 'service_virtualization_type': 'virtual-machine', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '73b076ee-b094-42da-a23f-85f67dd22a49'}, 'oper': 'CREATE', 'request-id': 'req-2ac08796-070b-489d-a4ea-d1466a30e66d', 'type': 'service_template', 'uuid': '73b076ee-b094-42da-a23f-85f67dd22a49'} 03/31/2025 02:15:18.515 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'netns-snat-template'] not found in DB: Unknown id: service_template default-domain:netns-snat-template 03/31/2025 02:15:18.516 7f0b598edcc0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:netns-snat-template url = https://10.0.0.17:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.17 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "netns-snat-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "source-nat", "image_name": null, "service_scaling": true, "interface_type": [{"service_interface_type": "right", "shared_ip": true, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": true, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "network-namespace", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "e808cb49d7ac4baab66085d471fe76a9", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "netns-snat-template"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:18.539 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'netns-snat-template'], 'obj_dict': {'display_name': 'netns-snat-template', 'fq_name': ['default-domain', 'netns-snat-template'], 'id_perms': {'created': '2025-03-31T02:15:18.532267', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:18.532267', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11387775810233506475, 'uuid_mslong': 9011719180571330053}}, 'parent_type': 'domain', 'parent_uuid': 'af15a304-5fa6-4208-ad60-abae44eb60f9', 'perms2': {'global_access': 0, 'owner': 'e808cb49d7ac4baab66085d471fe76a9', 'owner_access': 7, 'share': [{'tenant': 'domain:af15a304-5fa6-4208-ad60-abae44eb60f9', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': None, 'instance_data': None, 'interface_type': [{'service_interface_type': 'right', 'shared_ip': True, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': True, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': True, 'service_type': 'source-nat', 'service_virtualization_type': 'network-namespace', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '7d100ed9-3d87-4605-9e09-81c3aec11aab'}, 'oper': 'CREATE', 'request-id': 'req-1e10a84d-07cf-4311-9a0c-b37c1f2dd3e2', 'type': 'service_template', 'uuid': '7d100ed9-3d87-4605-9e09-81c3aec11aab'} 03/31/2025 02:15:18.600 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'haproxy-loadbalancer-template'] not found in DB: Unknown id: service_template default-domain:haproxy-loadbalancer-template 03/31/2025 02:15:18.602 7f0b5959dc18 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:haproxy-loadbalancer-template url = https://10.0.0.17:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.17 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "haproxy-loadbalancer-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "in-network-nat", "service_type": "loadbalancer", "image_name": null, "service_scaling": true, "interface_type": [{"service_interface_type": "right", "shared_ip": true, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": true, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "network-namespace", "availability_zone_enable": false, "vrouter_instance_type": null, "instance_data": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "e808cb49d7ac4baab66085d471fe76a9", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "haproxy-loadbalancer-template"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:18.630 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'haproxy-loadbalancer-template'], 'obj_dict': {'display_name': 'haproxy-loadbalancer-template', 'fq_name': ['default-domain', 'haproxy-loadbalancer-template'], 'id_perms': {'created': '2025-03-31T02:15:18.622669', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:18.622669', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12896974301153089741, 'uuid_mslong': 4512824269053642697}}, 'parent_type': 'domain', 'parent_uuid': 'af15a304-5fa6-4208-ad60-abae44eb60f9', 'perms2': {'global_access': 0, 'owner': 'e808cb49d7ac4baab66085d471fe76a9', 'owner_access': 7, 'share': [{'tenant': 'domain:af15a304-5fa6-4208-ad60-abae44eb60f9', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': None, 'instance_data': None, 'interface_type': [{'service_interface_type': 'right', 'shared_ip': True, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': True, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'in-network-nat', 'service_scaling': True, 'service_type': 'loadbalancer', 'service_virtualization_type': 'network-namespace', 'version': 1, 'vrouter_instance_type': None}, 'uuid': '3ea0c5c3-42b4-47c9-b2fb-41cbb99258cd'}, 'oper': 'CREATE', 'request-id': 'req-a3b21414-4c3a-4300-930f-613723752f57', 'type': 'service_template', 'uuid': '3ea0c5c3-42b4-47c9-b2fb-41cbb99258cd'} 03/31/2025 02:15:18.687 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'docker-template'] not found in DB: Unknown id: service_template default-domain:docker-template 03/31/2025 02:15:18.688 7f0b597feeb8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = service_template identifier_name = default-domain:docker-template url = https://10.0.0.17:8082/service-templates operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-svc-monitor remote_ip = 10.0.0.17 body = {"service-template": {"uuid": null, "fq_name": ["default-domain", "docker-template"], "parent_type": "domain", "service_template_properties": {"version": 1, "service_mode": "transparent", "service_type": "firewall", "image_name": "ubuntu", "service_scaling": false, "interface_type": [{"service_interface_type": "management", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "left", "shared_ip": false, "static_route_enable": false}, {"service_interface_type": "right", "shared_ip": false, "static_route_enable": false}], "flavor": null, "ordered_interfaces": true, "service_virtualization_type": "vrouter-instance", "availability_zone_enable": false, "vrouter_instance_type": "docker", "instance_data": "{\"command\":\"/bin/bash\"}"}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "e808cb49d7ac4baab66085d471fe76a9", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "docker-template"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:18.706 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'docker-template'], 'obj_dict': {'display_name': 'docker-template', 'fq_name': ['default-domain', 'docker-template'], 'id_perms': {'created': '2025-03-31T02:15:18.699089', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:18.699089', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13452603628822332632, 'uuid_mslong': 408367188951714043}}, 'parent_type': 'domain', 'parent_uuid': 'af15a304-5fa6-4208-ad60-abae44eb60f9', 'perms2': {'global_access': 0, 'owner': 'e808cb49d7ac4baab66085d471fe76a9', 'owner_access': 7, 'share': [{'tenant': 'domain:af15a304-5fa6-4208-ad60-abae44eb60f9', 'tenant_access': 5}]}, 'service_template_properties': {'availability_zone_enable': False, 'flavor': None, 'image_name': 'ubuntu', 'instance_data': '{"command":"/bin/bash"}', 'interface_type': [{'service_interface_type': 'management', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'left', 'shared_ip': False, 'static_route_enable': False}, {'service_interface_type': 'right', 'shared_ip': False, 'static_route_enable': False}], 'ordered_interfaces': True, 'service_mode': 'transparent', 'service_scaling': False, 'service_type': 'firewall', 'service_virtualization_type': 'vrouter-instance', 'version': 1, 'vrouter_instance_type': 'docker'}, 'uuid': '05aacfcb-7364-48fb-bab1-3fb9b81604d8'}, 'oper': 'CREATE', 'request-id': 'req-86648ba0-e6c1-464d-ab08-cc0d86e69c1f', 'type': 'service_template', 'uuid': '05aacfcb-7364-48fb-bab1-3fb9b81604d8'} 03/31/2025 02:15:18.728 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:19.479 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:15:19.480 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:15:19.480 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:15:19.480 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:15:19.480 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:15:19.480 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:15:19.480 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:15:19.480 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:15:19.480 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:15:19.480 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:15:19.480 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:15:19.481 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:15:19.482 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:15:19.482 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:15:19.482 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:15:19.482 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:15:19.482 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:15:19.482 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:15:19.482 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:15:19.482 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:15:19.482 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:15:19.482 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 03/31/2025 02:15:19.482 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:15:19.483 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:15:19.580 7f0b598e5c18 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'] not found in DB: Unknown id: config_node default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 03/31/2025 02:15:19.601 7f0b5938eeb8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 url = https://10.0.0.17:8082/config-nodes operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_config_node.py remote_ip = 10.0.0.17 body = {"config-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-juju-os-494-1"], "parent_type": "global-system-config", "config_node_ip_address": "10.0.0.17", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-juju-os-494-1"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:19.640 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'obj_dict': {'config_node_ip_address': '10.0.0.17', 'display_name': 'cn-jenkins-deploy-platform-juju-os-494-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'id_perms': {'created': '2025-03-31T02:15:19.620224', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:19.620224', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12761036259590971516, 'uuid_mslong': 14549044465539434888}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c9e896db-75d7-4988-b118-4ede02cca07c'}, 'oper': 'CREATE', 'request-id': 'req-52022a74-7d28-4831-9f7a-9501d598f8d3', 'type': 'config_node', 'uuid': 'c9e896db-75d7-4988-b118-4ede02cca07c'} 03/31/2025 02:15:19.641 7f0b5a7f2cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigNode:ContrailConfigTrace] map 03/31/2025 02:15:19.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:20.702 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:15:20.702 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:15:20.702 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:15:20.702 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:15:20.702 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:15:20.703 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:15:20.704 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:15:20.704 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:15:20.704 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:15:20.704 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:15:20.704 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:15:20.704 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:15:20.704 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:15:20.704 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:15:20.704 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:15:20.704 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:15:20.705 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:15:20.705 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:15:20.705 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:15:20.705 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:15:20.705 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:15:20.705 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 03/31/2025 02:15:20.705 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:15:20.705 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:15:20.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:20.821 7f0b59365ac8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'] not found in DB: Unknown id: config_database_node default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 03/31/2025 02:15:20.833 7f0b5959a2e8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = config_database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 url = https://10.0.0.17:8082/config-database-nodes operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_config_database_node.py remote_ip = 10.0.0.17 body = {"config-database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-juju-os-494-1"], "parent_type": "global-system-config", "config_database_node_ip_address": "10.0.0.17", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-juju-os-494-1"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:20.890 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'obj_dict': {'config_database_node_ip_address': '10.0.0.17', 'display_name': 'cn-jenkins-deploy-platform-juju-os-494-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'id_perms': {'created': '2025-03-31T02:15:20.865100', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:20.865100', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11957305701722566811, 'uuid_mslong': 10667208730384613528}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '940989d2-af85-4098-a5f0-e22ee5ed309b'}, 'oper': 'CREATE', 'request-id': 'req-0fa9e6db-6f12-4843-bf50-838ec49b2e1f', 'type': 'config_database_node', 'uuid': '940989d2-af85-4098-a5f0-e22ee5ed309b'} 03/31/2025 02:15:20.890 7f0b5a7f2cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectConfigDatabaseInfo:ContrailConfigTrace] map 03/31/2025 02:15:21.004 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_DNS_refs in resource network-ipam: 'virtual_DNS_refs' 03/31/2025 02:15:21.004 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource network-ipam: 'tag_refs' 03/31/2025 02:15:21.004 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource network-ipam: 'virtual_network_back_refs' 03/31/2025 02:15:21.005 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_router_back_refs in resource network-ipam: 'virtual_router_back_refs' 03/31/2025 02:15:21.005 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource network-ipam: 'instance_ip_back_refs' 03/31/2025 02:15:21.274 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:15:21.274 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:15:21.274 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:15:21.275 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:15:21.275 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:15:21.275 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:15:21.275 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:15:21.275 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:15:21.275 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:15:21.275 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:15:21.275 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:15:21.275 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:15:21.275 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:15:21.275 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:15:21.276 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:15:21.276 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:15:21.276 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:15:21.276 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:15:21.276 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:15:21.276 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:15:21.276 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:15:21.276 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:15:21.276 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:15:21.276 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:15:21.276 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:15:21.277 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:15:21.277 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:15:21.277 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:15:21.277 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:15:21.277 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:15:21.277 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:15:21.277 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:15:21.277 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 03/31/2025 02:15:21.277 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:15:21.278 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:15:21.375 7f0b5959a2e8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'] not found in DB: Unknown id: analytics_snmp_node default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 03/31/2025 02:15:21.380 7f0b5934a978 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_snmp_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 url = https://10.0.0.17:8082/analytics-snmp-nodes operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_analytics_snmp_node.py remote_ip = 10.0.0.17 body = {"analytics-snmp-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-juju-os-494-1"], "parent_type": "global-system-config", "analytics_snmp_node_ip_address": "10.0.0.17", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-juju-os-494-1"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:21.395 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'obj_dict': {'analytics_snmp_node_ip_address': '10.0.0.17', 'display_name': 'cn-jenkins-deploy-platform-juju-os-494-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'id_perms': {'created': '2025-03-31T02:15:21.389519', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:21.389519', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12070000015381377783, 'uuid_mslong': 6005397789573727639}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '53577578-be75-4597-a781-41107bc432f7'}, 'oper': 'CREATE', 'request-id': 'req-3e0a6b3f-0768-475e-913a-5dcf067dc1c1', 'type': 'analytics_snmp_node', 'uuid': '53577578-be75-4597-a781-41107bc432f7'} 03/31/2025 02:15:21.396 7f0b5a7f2cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectAnalyticsSNMPInfo:ContrailConfigTrace] map 03/31/2025 02:15:21.612 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'] not found in DB: Unknown id: access_control_list default-domain:default-project:__no_rule__:ingress-access-control-list 03/31/2025 02:15:21.619 7f0b592a48d0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:ingress-access-control-list url = https://10.0.0.17:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "__no_rule__", "ingress-access-control-list"], "parent_type": "security-group", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "ingress-access-control-list"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:21.647 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'ingress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'id_perms': {'created': '2025-03-31T02:15:21.634177', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:21.634177', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10127507956198847587, 'uuid_mslong': 15927913963893572198}}, 'parent_type': 'security-group', 'parent_uuid': 'd2359117-30fb-4bcd-a6bd-3469d720740e', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'dd0b515c-46dc-4266-8c8c-22d40d2cb463'}, 'oper': 'CREATE', 'request-id': 'req-19e2d8e6-0ac9-4f22-b4a8-e9980e2802f8', 'type': 'access_control_list', 'uuid': 'dd0b515c-46dc-4266-8c8c-22d40d2cb463'} 03/31/2025 02:15:21.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:21.740 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'] not found in DB: Unknown id: access_control_list default-domain:default-project:__no_rule__:egress-access-control-list 03/31/2025 02:15:21.747 7f0b57139630 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:egress-access-control-list url = https://10.0.0.17:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "__no_rule__", "egress-access-control-list"], "parent_type": "security-group", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "egress-access-control-list"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:21.767 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [], 'dynamic': None}, 'display_name': 'egress-access-control-list', 'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'id_perms': {'created': '2025-03-31T02:15:21.758281', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:21.758281', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10347465854837180141, 'uuid_mslong': 2179328767680726821}}, 'parent_type': 'security-group', 'parent_uuid': 'd2359117-30fb-4bcd-a6bd-3469d720740e', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '1e3e87f7-b666-4725-8f99-955f2b0dc6ed'}, 'oper': 'CREATE', 'request-id': 'req-25ceaa32-96f0-4d67-b472-39f72cfd76a1', 'type': 'access_control_list', 'uuid': '1e3e87f7-b666-4725-8f99-955f2b0dc6ed'} 03/31/2025 02:15:21.858 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000000'] not found in DB: Unknown id: route_target target:64512:8000000 03/31/2025 02:15:21.858 7f0b595c2ac8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000000 url = https://10.0.0.17:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000000"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "e808cb49d7ac4baab66085d471fe76a9", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000000"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:21.876 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000000'], 'obj_dict': {'display_name': 'target:64512:8000000', 'fq_name': ['target:64512:8000000'], 'id_perms': {'created': '2025-03-31T02:15:21.864957', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:21.864957', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12603755484295076151, 'uuid_mslong': 1489005304846893165}}, 'perms2': {'global_access': 0, 'owner': 'e808cb49d7ac4baab66085d471fe76a9', 'owner_access': 7, 'share': []}, 'uuid': '14aa026f-8812-406d-aee9-88d5dd4a1937'}, 'oper': 'CREATE', 'request-id': 'req-f2ea4f6d-55d8-49ec-9901-db0c0f77ef00', 'type': 'route_target', 'uuid': '14aa026f-8812-406d-aee9-88d5dd4a1937'} 03/31/2025 02:15:21.880 7f0b595c2438 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = b605e7a9-e7ea-4897-a1ae-ecb6573df558 object_type = routing_instance identifier_name = default-domain:default-project:dci-network:__default__ url = https://10.0.0.17:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"type": "routing-instance", "uuid": "b605e7a9-e7ea-4897-a1ae-ecb6573df558", "ref-type": "route-target", "ref-uuid": "14aa026f-8812-406d-aee9-88d5dd4a1937", "ref-fq-name": ["target:64512:8000000"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:21.891 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', '__default__'], 'oper': 'UPDATE', 'request-id': 'req-b96b8b7d-4625-4eef-b66c-edc717565622', 'type': 'routing_instance', 'uuid': 'b605e7a9-e7ea-4897-a1ae-ecb6573df558'} 03/31/2025 02:15:21.945 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000001'] not found in DB: Unknown id: route_target target:64512:8000001 03/31/2025 02:15:21.946 7f0b593918d0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000001 url = https://10.0.0.17:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000001"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "e808cb49d7ac4baab66085d471fe76a9", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000001"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:21.959 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000001'], 'obj_dict': {'display_name': 'target:64512:8000001', 'fq_name': ['target:64512:8000001'], 'id_perms': {'created': '2025-03-31T02:15:21.951291', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:21.951291', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11230688320785740223, 'uuid_mslong': 18403980660237223733}}, 'perms2': {'global_access': 0, 'owner': 'e808cb49d7ac4baab66085d471fe76a9', 'owner_access': 7, 'share': []}, 'uuid': 'ff6812e6-ea99-4735-9bdb-6b86678ff1bf'}, 'oper': 'CREATE', 'request-id': 'req-83e21517-4027-445f-a7ce-9f3dc0f7710a', 'type': 'route_target', 'uuid': 'ff6812e6-ea99-4735-9bdb-6b86678ff1bf'} 03/31/2025 02:15:22.010 7f0b59818048 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = cba33f29-5552-4c35-a2e5-4e40e5cee823 object_type = routing_instance identifier_name = default-domain:default-project:dci-network:dci-network url = https://10.0.0.17:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"type": "routing-instance", "uuid": "cba33f29-5552-4c35-a2e5-4e40e5cee823", "ref-type": "route-target", "ref-uuid": "ff6812e6-ea99-4735-9bdb-6b86678ff1bf", "ref-fq-name": ["target:64512:8000001"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:22.025 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'dci-network', 'dci-network'], 'oper': 'UPDATE', 'request-id': 'req-b1244ebd-c688-43c8-a744-06eb4767a561', 'type': 'routing_instance', 'uuid': 'cba33f29-5552-4c35-a2e5-4e40e5cee823'} 03/31/2025 02:15:22.081 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000002'] not found in DB: Unknown id: route_target target:64512:8000002 03/31/2025 02:15:22.082 7f0b593294e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000002 url = https://10.0.0.17:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000002"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "e808cb49d7ac4baab66085d471fe76a9", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000002"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:22.093 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000002'], 'obj_dict': {'display_name': 'target:64512:8000002', 'fq_name': ['target:64512:8000002'], 'id_perms': {'created': '2025-03-31T02:15:22.087911', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:22.087911', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10204744915359585052, 'uuid_mslong': 6344508486168167303}}, 'perms2': {'global_access': 0, 'owner': 'e808cb49d7ac4baab66085d471fe76a9', 'owner_access': 7, 'share': []}, 'uuid': '580c38da-8081-4387-8d9e-896ef77b531c'}, 'oper': 'CREATE', 'request-id': 'req-e36e148f-ecd7-4814-891f-23c85d0cfcee', 'type': 'route_target', 'uuid': '580c38da-8081-4387-8d9e-896ef77b531c'} 03/31/2025 02:15:22.154 7f0b5959e978 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 992fb5d0-4feb-4a33-ad69-6359940b7ef6 object_type = routing_instance identifier_name = default-domain:default-project:default-virtual-network:default-virtual-network url = https://10.0.0.17:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"type": "routing-instance", "uuid": "992fb5d0-4feb-4a33-ad69-6359940b7ef6", "ref-type": "route-target", "ref-uuid": "580c38da-8081-4387-8d9e-896ef77b531c", "ref-fq-name": ["target:64512:8000002"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:22.178 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'default-virtual-network', 'default-virtual-network'], 'oper': 'UPDATE', 'request-id': 'req-20ab8157-52d5-4961-bb73-c3e7e3b5968f', 'type': 'routing_instance', 'uuid': '992fb5d0-4feb-4a33-ad69-6359940b7ef6'} 03/31/2025 02:15:22.237 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['target:64512:8000003'] not found in DB: Unknown id: route_target target:64512:8000003 03/31/2025 02:15:22.241 7f0b593654e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = route_target identifier_name = target:64512:8000003 url = https://10.0.0.17:8082/route-targets operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"route-target": {"uuid": null, "fq_name": ["target:64512:8000003"], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "e808cb49d7ac4baab66085d471fe76a9", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "target:64512:8000003"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:22.267 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['target:64512:8000003'], 'obj_dict': {'display_name': 'target:64512:8000003', 'fq_name': ['target:64512:8000003'], 'id_perms': {'created': '2025-03-31T02:15:22.248059', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:22.248059', 'permissions': {'group': 'cloud-admin-group', 'group_access': 7, 'other_access': 7, 'owner': 'cloud-admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9701717668651750712, 'uuid_mslong': 11371385079478897633}}, 'perms2': {'global_access': 0, 'owner': 'e808cb49d7ac4baab66085d471fe76a9', 'owner_access': 7, 'share': []}, 'uuid': '9dcf467b-4a84-4be1-86a3-6cd1571e5538'}, 'oper': 'CREATE', 'request-id': 'req-16e87337-d7c6-4d44-82d4-182526db77b5', 'type': 'route_target', 'uuid': '9dcf467b-4a84-4be1-86a3-6cd1571e5538'} 03/31/2025 02:15:22.271 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:15:22.271 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:15:22.271 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:15:22.271 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:15:22.271 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:15:22.271 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:15:22.271 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:15:22.272 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:15:22.273 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:15:22.273 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:15:22.273 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:15:22.273 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:15:22.273 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:15:22.273 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:15:22.273 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:15:22.273 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:15:22.273 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:15:22.273 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:15:22.273 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:15:22.274 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:15:22.274 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:15:22.274 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 03/31/2025 02:15:22.274 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:15:22.274 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:15:22.308 7f0b5a1bb198 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = e10ef3f4-720e-40a6-af67-7f24c5388536 object_type = routing_instance identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = https://10.0.0.17:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"type": "routing-instance", "uuid": "e10ef3f4-720e-40a6-af67-7f24c5388536", "ref-type": "route-target", "ref-uuid": "9dcf467b-4a84-4be1-86a3-6cd1571e5538", "ref-fq-name": ["target:64512:8000003"], "operation": "ADD", "attr": {"import_export": null}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:22.319 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-683092c8-9b3f-43e6-9066-05758c9ffead', 'type': 'routing_instance', 'uuid': 'e10ef3f4-720e-40a6-af67-7f24c5388536'} 03/31/2025 02:15:22.382 7f0b595774e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'] not found in DB: Unknown id: database_node default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 03/31/2025 02:15:22.387 7f0b59391eb8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = database_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 url = https://10.0.0.17:8082/database-nodes operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_database_node.py remote_ip = 10.0.0.17 body = {"database-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-juju-os-494-1"], "parent_type": "global-system-config", "database_node_ip_address": "10.0.0.17", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-juju-os-494-1"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:22.405 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'obj_dict': {'database_node_ip_address': '10.0.0.17', 'display_name': 'cn-jenkins-deploy-platform-juju-os-494-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'id_perms': {'created': '2025-03-31T02:15:22.399696', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:22.399696', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12111194121331307240, 'uuid_mslong': 17784921464622632095}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'f6d0bbd7-f10d-4c9f-a813-9ae2afc072e8'}, 'oper': 'CREATE', 'request-id': 'req-2f8b9cc0-4ebb-4739-98d9-277898e005ad', 'type': 'database_node', 'uuid': 'f6d0bbd7-f10d-4c9f-a813-9ae2afc072e8'} 03/31/2025 02:15:22.408 7f0b5a7f2cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectDatabaseInfo:ContrailConfigTrace] map 03/31/2025 02:15:22.440 7f0b5deb20f0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'] not found in DB: Unknown id: access_control_list default-domain:default-project:ip-fabric:ip-fabric 03/31/2025 02:15:22.445 7f0b5936bf60 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = access_control_list identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = https://10.0.0.17:8082/access-control-lists operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"access-control-list": {"uuid": null, "fq_name": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "parent_type": "virtual-network", "access_control_list_entries": {"dynamic": false, "acl_rule": [{"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "deny", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000002", "direction": "<>"}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "ip-fabric"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:22.525 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'obj_dict': {'access_control_list_entries': {'acl_rule': [{'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'deny'}, 'direction': '<>', 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'default-domain:default-project:ip-fabric'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000002'}, {'action_list': {'alert': False, 'apply_service': [], 'assign_routing_instance': None, 'gateway_name': None, 'host_based_service': False, 'log': False, 'mirror_to': None, 'qos_action': None, 'service_properties': None, 'simple_action': 'pass'}, 'direction': None, 'match_condition': {'dst_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'dst_port': {'end_port': 65535, 'start_port': 0}, 'ethertype': None, 'protocol': 'any', 'src_address': {'network_policy': None, 'security_group': None, 'subnet': None, 'subnet_list': [], 'virtual_network': 'any'}, 'src_port': {'end_port': 65535, 'start_port': 0}}, 'rule_uuid': '00000000-0000-0000-0000-100000000001'}], 'dynamic': False}, 'display_name': 'ip-fabric', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'id_perms': {'created': '2025-03-31T02:15:22.461505', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:22.461505', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13439486164779662646, 'uuid_mslong': 6339529874178458562}}, 'parent_type': 'virtual-network', 'parent_uuid': '3da78d9c-3d98-4934-a990-647a4cdbfe7f', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '57fa88d5-1162-43c2-ba82-a5761f77fd36'}, 'oper': 'CREATE', 'request-id': 'req-fad57d89-3839-4bd7-a1f6-e3cea84c1547', 'type': 'access_control_list', 'uuid': '57fa88d5-1162-43c2-ba82-a5761f77fd36'} 03/31/2025 02:15:22.564 7f0b59391eb8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = dd0b515c-46dc-4266-8c8c-22d40d2cb463 object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:ingress-access-control-list url = https://10.0.0.17:8082/access-control-list/dd0b515c-46dc-4266-8c8c-22d40d2cb463 operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"access-control-list": {"uuid": "dd0b515c-46dc-4266-8c8c-22d40d2cb463", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "access_control_list_hash": -782339169199063927}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:22.580 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'ingress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-eb36c79f-f3ce-47b6-a80e-78e2ff620d3d', 'type': 'access_control_list', 'uuid': 'dd0b515c-46dc-4266-8c8c-22d40d2cb463'} 03/31/2025 02:15:22.624 7f0b59365048 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 1e3e87f7-b666-4725-8f99-955f2b0dc6ed object_type = access_control_list identifier_name = default-domain:default-project:__no_rule__:egress-access-control-list url = https://10.0.0.17:8082/access-control-list/1e3e87f7-b666-4725-8f99-955f2b0dc6ed operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"access-control-list": {"uuid": "1e3e87f7-b666-4725-8f99-955f2b0dc6ed", "access_control_list_entries": {"dynamic": null, "acl_rule": []}, "access_control_list_hash": -782339169199063927}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:22.638 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', '__no_rule__', 'egress-access-control-list'], 'oper': 'UPDATE', 'request-id': 'req-a1a2f77e-63dd-415b-b2b8-09c1fdfca480', 'type': 'access_control_list', 'uuid': '1e3e87f7-b666-4725-8f99-955f2b0dc6ed'} 03/31/2025 02:15:22.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:23.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:23.747 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 03/31/2025 02:15:23.747 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 03/31/2025 02:15:23.747 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 03/31/2025 02:15:23.747 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 03/31/2025 02:15:23.747 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 03/31/2025 02:15:23.747 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 03/31/2025 02:15:23.747 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 03/31/2025 02:15:23.747 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 03/31/2025 02:15:23.795 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-juju-os-494-1'] not found in DB: Unknown id: bgp_router default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-juju-os-494-1 03/31/2025 02:15:23.800 7f0b5959df60 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = bgp_router identifier_name = default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-juju-os-494-1 url = https://10.0.0.17:8082/bgp-routers operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.17 body = {"bgp-router": {"uuid": null, "fq_name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-juju-os-494-1"], "parent_type": "routing-instance", "bgp_router_parameters": {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.0.0.17", "address": "10.0.0.17", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-juju-os-494-1"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:23.824 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'obj_dict': {'bgp_router_parameters': {'address': '10.0.0.17', 'address_families': {'family': ['route-target', 'inet-vpn', 'e-vpn', 'erm-vpn', 'inet6-vpn']}, 'admin_down': False, 'auth_data': None, 'autonomous_system': 64512, 'cluster_id': None, 'gateway_address': None, 'hold_time': 0, 'identifier': '10.0.0.17', 'ipv6_gateway_address': None, 'local_autonomous_system': None, 'port': 179, 'router_type': 'control-node', 'source_port': None, 'vendor': 'contrail'}, 'display_name': 'cn-jenkins-deploy-platform-juju-os-494-1', 'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'id_perms': {'created': '2025-03-31T02:15:23.817579', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:23.817579', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10044122602131072710, 'uuid_mslong': 11875047628104616913}}, 'parent_type': 'routing-instance', 'parent_uuid': 'd7ea1fdf-c27f-4008-8f8f-5741e27e1e75', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a4cca4e6-ac05-4bd1-8b63-e44a647ea6c6'}, 'oper': 'CREATE', 'request-id': 'req-baf223ca-9c92-49d4-a1b3-20269b7221a1', 'type': 'bgp_router', 'uuid': 'a4cca4e6-ac05-4bd1-8b63-e44a647ea6c6'} 03/31/2025 02:15:23.825 7f0b5a7f2cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectBgpRouter:ContrailConfigTrace] map 03/31/2025 02:15:23.877 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zone_refs in resource bgp-router: 'control_node_zone_refs' 03/31/2025 02:15:23.878 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource bgp-router: 'sub_cluster_refs' 03/31/2025 02:15:23.878 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource bgp-router: 'bgp_router_refs' 03/31/2025 02:15:23.878 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource bgp-router: 'tag_refs' 03/31/2025 02:15:23.878 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_system_config_back_refs in resource bgp-router: 'global_system_config_back_refs' 03/31/2025 02:15:23.878 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource bgp-router: 'physical_router_back_refs' 03/31/2025 02:15:23.878 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource bgp-router: 'virtual_machine_interface_back_refs' 03/31/2025 02:15:23.878 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_as_a_service_back_refs in resource bgp-router: 'bgp_as_a_service_back_refs' 03/31/2025 02:15:23.878 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_back_refs in resource bgp-router: 'bgp_router_back_refs' 03/31/2025 02:15:23.979 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:15:23.979 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:15:23.980 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:15:23.981 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:15:23.981 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:15:23.981 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:15:23.981 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:15:23.981 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:15:23.981 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:15:23.981 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:15:23.981 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:15:23.981 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:15:23.981 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:15:23.981 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:15:23.982 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:15:23.982 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:15:23.982 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:15:23.982 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:15:23.982 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:15:23.982 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:15:23.982 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:15:23.982 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource global-system-config: 'bgp_router_refs' 03/31/2025 02:15:23.982 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:15:23.982 7f0b5936e198 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:15:24.036 7f0b5959d6d8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 0d5441e0-0a93-4574-96f2-d4e0e141b4e1 object_type = global_system_config identifier_name = default-global-system-config url = https://10.0.0.17:8082/ref-relax-for-delete operation = ref-relax-for-delete useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.17 body = {"uuid": "0d5441e0-0a93-4574-96f2-d4e0e141b4e1", "ref-uuid": "a4cca4e6-ac05-4bd1-8b63-e44a647ea6c6"} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:24.089 7f0b595c0c18 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 0d5441e0-0a93-4574-96f2-d4e0e141b4e1 object_type = global_system_config identifier_name = default-global-system-config url = https://10.0.0.17:8082/ref-update operation = ref-update useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.17 body = {"type": "global-system-config", "uuid": "0d5441e0-0a93-4574-96f2-d4e0e141b4e1", "ref-type": "bgp-router", "ref-uuid": "a4cca4e6-ac05-4bd1-8b63-e44a647ea6c6", "ref-fq-name": ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-juju-os-494-1"], "operation": "ADD", "attr": null} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:24.099 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-bd68c9ae-a639-4c62-9c4e-79b9deaba4d0', 'type': 'global_system_config', 'uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1'} 03/31/2025 02:15:24.326 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:15:24.326 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:15:24.326 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:15:24.326 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:15:24.326 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:15:24.326 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:15:24.326 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:15:24.327 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:15:24.327 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:15:24.327 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:15:24.327 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:15:24.327 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:15:24.327 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:15:24.327 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:15:24.327 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:15:24.327 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:15:24.328 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:15:24.328 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:15:24.328 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:15:24.328 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:15:24.328 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:15:24.328 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:15:24.328 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:15:24.328 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:15:24.328 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:15:24.329 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:15:24.329 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:15:24.329 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:15:24.329 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:15:24.329 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:15:24.329 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:15:24.329 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:15:24.329 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:15:24.329 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:15:24.427 7f0b598e08d0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'] not found in DB: Unknown id: analytics_node default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 03/31/2025 02:15:24.432 7f0b5a809198 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = analytics_node identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 url = https://10.0.0.17:8082/analytics-nodes operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_analytics_node.py remote_ip = 10.0.0.17 body = {"analytics-node": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-juju-os-494-1"], "parent_type": "global-system-config", "analytics_node_ip_address": "10.0.0.17", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-juju-os-494-1"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:24.457 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'obj_dict': {'analytics_node_ip_address': '10.0.0.17', 'display_name': 'cn-jenkins-deploy-platform-juju-os-494-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'id_perms': {'created': '2025-03-31T02:15:24.446408', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:24.446408', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10378292755966028175, 'uuid_mslong': 16495268877807076641}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'e4eaf7a6-103f-4521-9007-1a46665d698f'}, 'oper': 'CREATE', 'request-id': 'req-e56a59d7-589f-433c-9b4d-d33cece9ad03', 'type': 'analytics_node', 'uuid': 'e4eaf7a6-103f-4521-9007-1a46665d698f'} 03/31/2025 02:15:24.457 7f0b5a7f2cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectCollectorInfo:ContrailConfigTrace] map 03/31/2025 02:15:24.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:25.491 7f0b595c7438 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 03/31/2025 02:15:25.491 7f0b595c7438 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 03/31/2025 02:15:25.491 7f0b595c7438 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 03/31/2025 02:15:25.596 7f0b5959a2e8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 1e3d2dbd-97fa-4e55-8bd1-1e895383b50a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = https://10.0.0.17:8082/global-vrouter-config/1e3d2dbd-97fa-4e55-8bd1-1e895383b50a operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_linklocal.py remote_ip = 10.0.0.17 body = {"global-vrouter-config": {"uuid": "1e3d2dbd-97fa-4e55-8bd1-1e895383b50a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "linklocal_services": {"linklocal_service_entry": [{"linklocal_service_name": "metadata", "linklocal_service_ip": "169.254.169.254", "linklocal_service_port": 80, "ip_fabric_DNS_service_name": "", "ip_fabric_service_port": 8775, "ip_fabric_service_ip": ["127.0.0.1"]}]}}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:25.605 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-56128ccb-102f-43fb-8cf9-cd2c69de3b94', 'type': 'global_vrouter_config', 'uuid': '1e3d2dbd-97fa-4e55-8bd1-1e895383b50a'} 03/31/2025 02:15:25.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:26.373 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-conf-elements-inconsistency'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-conf-elements-inconsistency 03/31/2025 02:15:26.377 7f0b5959df60 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-conf-elements-inconsistency url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-conf-elements-inconsistency"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements", "operation": "size==", "operand2": {"json_value": "1"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "ContrailConfig has a list of elements - it should be a dict. Controllers have inconsistency in their data.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-conf-elements-inconsistency"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:26.393 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements', 'operand2': {'json_value': '1'}, 'operation': 'size=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-elements-inconsistency', 'fq_name': ['default-global-system-config', 'system-defined-conf-elements-inconsistency'], 'id_perms': {'created': '2025-03-31T02:15:26.387631', 'creator': None, 'description': 'ContrailConfig has a list of ' 'elements - it should be a dict. ' 'Controllers have inconsistency in ' 'their data.', 'enable': True, 'last_modified': '2025-03-31T02:15:26.387631', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11824390800763475337, 'uuid_mslong': 2795755916297323579}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '26cc8531-7d99-443b-a418-acc8f4e75589', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-ed50ec06-2167-4648-97fe-b302ebfe2db1', 'type': 'alarm', 'uuid': '26cc8531-7d99-443b-a418-acc8f4e75589'} 03/31/2025 02:15:26.399 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-address-mismatch-compute'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-address-mismatch-compute 03/31/2025 02:15:26.403 7f0b5ec8fe10 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-compute url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-address-mismatch-compute"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_ip_address", "operation": "!=", "operand2": {"uve_attribute": "VrouterAgent.control_ip"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Compute Node IP Address mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-address-mismatch-compute"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:26.442 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_ip_address', 'operand2': {'uve_attribute': 'VrouterAgent.control_ip'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-compute', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-compute'], 'id_perms': {'created': '2025-03-31T02:15:26.415253', 'creator': None, 'description': 'Compute Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-03-31T02:15:26.415253', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12042074166954571496, 'uuid_mslong': 6928701809260579503}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6027b195-ef95-4aaf-a71e-0aa7383c36e8', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-aa3dc3dd-95c6-44ea-8438-3d7a506c0b99', 'type': 'alarm', 'uuid': '6027b195-ef95-4aaf-a71e-0aa7383c36e8'} 03/31/2025 02:15:26.483 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-address-mismatch-control'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-address-mismatch-control 03/31/2025 02:15:26.487 7f0b5959ed68 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-address-mismatch-control url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-address-mismatch-control"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.bgp_router_parameters.address", "operation": "not in", "operand2": {"uve_attribute": "BgpRouterState.bgp_router_ip_list"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Control Node IP Address mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-address-mismatch-control"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:26.505 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.bgp_router_parameters.address', 'operand2': {'uve_attribute': 'BgpRouterState.bgp_router_ip_list'}, 'operation': 'not ' 'in'}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-address-mismatch-control', 'fq_name': ['default-global-system-config', 'system-defined-address-mismatch-control'], 'id_perms': {'created': '2025-03-31T02:15:26.498013', 'creator': None, 'description': 'Control Node IP Address mismatch.', 'enable': True, 'last_modified': '2025-03-31T02:15:26.498013', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9313572778354239045, 'uuid_mslong': 15858663041376011575}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'dc154a03-69d5-4537-8140-7518b38d6645', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-e31cd8b5-4c5a-4ee4-9b53-e45345c7bf06', 'type': 'alarm', 'uuid': 'dc154a03-69d5-4537-8140-7518b38d6645'} 03/31/2025 02:15:26.551 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-bgp-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-bgp-connectivity 03/31/2025 02:15:26.555 7f0b5998d2e8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-bgp-connectivity url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-bgp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_bgp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_bgp_peer"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "BGP peer mismatch. Not enough BGP peers are up.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-bgp-connectivity"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:26.574 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_bgp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_bgp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-bgp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-bgp-connectivity'], 'id_perms': {'created': '2025-03-31T02:15:26.566400', 'creator': None, 'description': 'BGP peer mismatch. Not enough BGP ' 'peers are up.', 'enable': True, 'last_modified': '2025-03-31T02:15:26.566400', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12143534669654242620, 'uuid_mslong': 3478817349111465422}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '30473df2-8c41-45ce-a886-80715e61b53c', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-2ba045e4-7ec6-4458-89dc-4e59af96f9bf', 'type': 'alarm', 'uuid': '30473df2-8c41-45ce-a886-80715e61b53c'} 03/31/2025 02:15:26.590 7f0b592b4978 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:15:26.619 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-conf-incorrect'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-conf-incorrect 03/31/2025 02:15:26.623 7f0b5959dac8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-conf-incorrect url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-conf-incorrect"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "ContrailConfig missing or incorrect. Configuration pushed to Ifmap as ContrailConfig is missing/incorrect.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-conf-incorrect"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:26.640 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-conf-incorrect', 'fq_name': ['default-global-system-config', 'system-defined-conf-incorrect'], 'id_perms': {'created': '2025-03-31T02:15:26.633044', 'creator': None, 'description': 'ContrailConfig missing or ' 'incorrect. Configuration pushed to ' 'Ifmap as ContrailConfig is ' 'missing/incorrect.', 'enable': True, 'last_modified': '2025-03-31T02:15:26.633044', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13249334918619076279, 'uuid_mslong': 15140899578280955235}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'd21f47f2-4f4c-4d63-b7df-17e9167816b7', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-c33488b4-3b89-42a7-8fa2-851e2ccbaa1b', 'type': 'alarm', 'uuid': 'd21f47f2-4f4c-4d63-b7df-17e9167816b7'} 03/31/2025 02:15:26.647 7f0b592beb70 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:15:26.683 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-disk-usage-high'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-disk-usage-high 03/31/2025 02:15:26.688 7f0b5ec8fe10 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-high url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-disk-usage-high"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": "range", "operand2": {"json_value": "[70, 90]"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Disk usage crosses high threshold limit.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-disk-usage-high"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:26.705 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '[70, ' '90]'}, 'operation': 'range', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-disk-usage-high', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-high'], 'id_perms': {'created': '2025-03-31T02:15:26.698784', 'creator': None, 'description': 'Disk usage crosses high threshold ' 'limit.', 'enable': True, 'last_modified': '2025-03-31T02:15:26.698784', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13588284882863692473, 'uuid_mslong': 7568459836537392601}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '6908921f-359f-4dd9-bc93-491b6de69ab9', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-996c9108-9be9-4c84-ac23-db07e640c3be', 'type': 'alarm', 'uuid': '6908921f-359f-4dd9-bc93-491b6de69ab9'} 03/31/2025 02:15:26.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:26.751 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-disk-usage-critical'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-disk-usage-critical 03/31/2025 02:15:26.755 7f0b5959dac8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-disk-usage-critical url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-disk-usage-critical"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.disk_usage_info.*.percentage_partition_space_used", "operation": ">", "operand2": {"json_value": "90"}, "variables": ["NodeStatus.disk_usage_info.__key"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Disk usage crosses critical threshold limit.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-disk-usage-critical"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:26.778 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.disk_usage_info.*.percentage_partition_space_used', 'operand2': {'json_value': '90'}, 'operation': '>', 'variables': ['NodeStatus.disk_usage_info.__key']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-disk-usage-critical', 'fq_name': ['default-global-system-config', 'system-defined-disk-usage-critical'], 'id_perms': {'created': '2025-03-31T02:15:26.769943', 'creator': None, 'description': 'Disk usage crosses critical ' 'threshold limit.', 'enable': True, 'last_modified': '2025-03-31T02:15:26.769943', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10747137106908478361, 'uuid_mslong': 7545478664436139798}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '68b6ecde-1e01-4716-9525-804209b2fb99', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-15637184-884c-470f-8977-b5bb326c469a', 'type': 'alarm', 'uuid': '68b6ecde-1e01-4716-9525-804209b2fb99'} 03/31/2025 02:15:26.823 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-node-status'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-node-status 03/31/2025 02:15:26.828 7f0b59405eb8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-node-status url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-node-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Node Failure. NodeStatus UVE not present.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-node-status"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:26.849 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-node-status', 'fq_name': ['default-global-system-config', 'system-defined-node-status'], 'id_perms': {'created': '2025-03-31T02:15:26.840121', 'creator': None, 'description': 'Node Failure. NodeStatus UVE not ' 'present.', 'enable': True, 'last_modified': '2025-03-31T02:15:26.840121', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13006352620774726339, 'uuid_mslong': 13396966605774407175}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'b9eb9625-d4f4-4607-b47f-d8cc48bf9ac3', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-a7f258e5-c7f1-4a61-a537-df90c003113f', 'type': 'alarm', 'uuid': 'b9eb9625-d4f4-4607-b47f-d8cc48bf9ac3'} 03/31/2025 02:15:26.891 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-partial-sysinfo'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-partial-sysinfo 03/31/2025 02:15:26.896 7f0b592c0588 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-partial-sysinfo url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-partial-sysinfo"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.build_info", "operation": "==", "operand2": {"json_value": "null"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "System Info Incomplete.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-partial-sysinfo"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:26.921 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.build_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-partial-sysinfo', 'fq_name': ['default-global-system-config', 'system-defined-partial-sysinfo'], 'id_perms': {'created': '2025-03-31T02:15:26.912639', 'creator': None, 'description': 'System Info Incomplete.', 'enable': True, 'last_modified': '2025-03-31T02:15:26.912639', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11735771544900595085, 'uuid_mslong': 11827768804162158665}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'a424ad0f-a988-4049-a2dd-d60a92efed8d', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-c354fddb-3dca-468b-97b7-76dd8f828661', 'type': 'alarm', 'uuid': 'a424ad0f-a988-4049-a2dd-d60a92efed8d'} 03/31/2025 02:15:26.969 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-process-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-process-connectivity 03/31/2025 02:15:26.974 7f0b59403c18 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-process-connectivity url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-process-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_status", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_status.state", "operation": "!=", "operand2": {"json_value": "\"Functional\""}, "variables": ["NodeStatus.process_status.module_id", "NodeStatus.process_status.instance_id"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Process(es) reporting as non-functional.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-process-connectivity"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:26.995 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_status', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_status.state', 'operand2': {'json_value': '"Functional"'}, 'operation': '!=', 'variables': ['NodeStatus.process_status.module_id', 'NodeStatus.process_status.instance_id']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-process-connectivity'], 'id_perms': {'created': '2025-03-31T02:15:26.987444', 'creator': None, 'description': 'Process(es) reporting as ' 'non-functional.', 'enable': True, 'last_modified': '2025-03-31T02:15:26.987444', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11146195899856800742, 'uuid_mslong': 17270889996190566830}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'efae86f2-49c9-49ae-9aaf-3e1da7e14be6', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-efbd095b-1a14-4728-82dd-26c960cf84b6', 'type': 'alarm', 'uuid': 'efae86f2-49c9-49ae-9aaf-3e1da7e14be6'} 03/31/2025 02:15:27.040 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-process-status'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-process-status 03/31/2025 02:15:27.046 7f0b59326d68 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-process-status url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-process-status"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.process_info", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "NodeStatus.process_info.process_state", "operation": "!=", "operand2": {"json_value": "\"PROCESS_STATE_RUNNING\""}, "variables": ["NodeStatus.process_info.process_name"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Process Failure.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-process-status"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:27.278 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.process_info', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'NodeStatus.process_info.process_state', 'operand2': {'json_value': '"PROCESS_STATE_RUNNING"'}, 'operation': '!=', 'variables': ['NodeStatus.process_info.process_name']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-process-status', 'fq_name': ['default-global-system-config', 'system-defined-process-status'], 'id_perms': {'created': '2025-03-31T02:15:27.271450', 'creator': None, 'description': 'Process Failure.', 'enable': True, 'last_modified': '2025-03-31T02:15:27.271450', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11308765916160519600, 'uuid_mslong': 1234713806297907430}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '112295aa-3fa8-40e6-9cf0-ceaf30caedb0', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-ea1cc910-a067-4560-a24d-ef9bc2cad440', 'type': 'alarm', 'uuid': '112295aa-3fa8-40e6-9cf0-ceaf30caedb0'} 03/31/2025 02:15:27.324 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-prouter-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-prouter-connectivity 03/31/2025 02:15:27.329 7f0b5932af60 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-connectivity url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-prouter-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.connected_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Prouter connectivity to controlling tor agent does not exist we look for non-empty value for connected_agent_list", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-prouter-connectivity"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:27.399 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.connected_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-connectivity'], 'id_perms': {'created': '2025-03-31T02:15:27.388982', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tor agent does not exist we look ' 'for non-empty value for ' 'connected_agent_list', 'enable': True, 'last_modified': '2025-03-31T02:15:27.388982', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9772858218520621087, 'uuid_mslong': 1657443375306457942}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '17006bf7-e477-4356-87a0-2ac55b2db01f', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-9e52bed0-1b55-4d24-a41a-ed9ceda43fce', 'type': 'alarm', 'uuid': '17006bf7-e477-4356-87a0-2ac55b2db01f'} 03/31/2025 02:15:27.446 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-prouter-tsn-connectivity 03/31/2025 02:15:27.454 7f0b594036d8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-prouter-tsn-connectivity url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-prouter-tsn-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["prouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "1"}}, {"operand1": "ProuterData.gateway_mode", "operation": "!=", "operand2": {"json_value": "\"SERVER\""}}]}, {"and_list": [{"operand1": "ContrailConfig.elements.virtual_router_refs", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "ProuterData.tsn_agent_list", "operation": "size!=", "operand2": {"json_value": "0"}}, {"operand1": "ProuterData.gateway_mode", "operation": "==", "operand2": {"json_value": "\"SERVER\""}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Prouter connectivity to controlling tsn agent does not exist we look for non-empty value for tsn_agent_list", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-prouter-tsn-connectivity"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:27.536 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '1'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '!='}]}, {'and_list': [{'operand1': 'ContrailConfig.elements.virtual_router_refs', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'ProuterData.tsn_agent_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}, {'operand1': 'ProuterData.gateway_mode', 'operand2': {'json_value': '"SERVER"'}, 'operation': '=='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-prouter-tsn-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-prouter-tsn-connectivity'], 'id_perms': {'created': '2025-03-31T02:15:27.479091', 'creator': None, 'description': 'Prouter connectivity to controlling ' 'tsn agent does not exist we look ' 'for non-empty value for ' 'tsn_agent_list', 'enable': True, 'last_modified': '2025-03-31T02:15:27.479091', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9577759536017800359, 'uuid_mslong': 10578677128556135903}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '92cf02cc-d923-45df-84eb-098d88737ca7', 'uve_keys': {'uve_key': ['prouter']}}, 'oper': 'CREATE', 'request-id': 'req-f144e8f6-b46c-42c4-8358-93b69a76a1ba', 'type': 'alarm', 'uuid': '92cf02cc-d923-45df-84eb-098d88737ca7'} 03/31/2025 02:15:27.577 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-storage-cluster-state'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-storage-cluster-state 03/31/2025 02:15:27.583 7f0b592c0828 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-storage-cluster-state url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-storage-cluster-state"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["storage-cluster"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "StorageCluster.info_stats.status", "operation": "!=", "operand2": {"json_value": "0"}, "variables": ["StorageCluster.info_stats.health_summary"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Storage Cluster warning/errors.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-storage-cluster-state"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:27.605 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'StorageCluster.info_stats.status', 'operand2': {'json_value': '0'}, 'operation': '!=', 'variables': ['StorageCluster.info_stats.health_summary']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-storage-cluster-state', 'fq_name': ['default-global-system-config', 'system-defined-storage-cluster-state'], 'id_perms': {'created': '2025-03-31T02:15:27.597098', 'creator': None, 'description': 'Storage Cluster warning/errors.', 'enable': True, 'last_modified': '2025-03-31T02:15:27.597098', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9308612904492263808, 'uuid_mslong': 17246029238992323152}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'ef563438-3023-4250-812e-d61e13e0e980', 'uve_keys': {'uve_key': ['storage-cluster']}}, 'oper': 'CREATE', 'request-id': 'req-6a63684a-eac9-4bf1-b82d-b032f9db59e0', 'type': 'alarm', 'uuid': 'ef563438-3023-4250-812e-d61e13e0e980'} 03/31/2025 02:15:27.652 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-interface'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-interface 03/31/2025 02:15:27.658 7f0b59322eb8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-interface url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-interface"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.down_interface_count", "operation": ">=", "operand2": {"json_value": "1"}, "variables": ["VrouterAgent.error_intf_list", "VrouterAgent.no_config_intf_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Vrouter interface(s) down.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-interface"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:27.687 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.down_interface_count', 'operand2': {'json_value': '1'}, 'operation': '>=', 'variables': ['VrouterAgent.error_intf_list', 'VrouterAgent.no_config_intf_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-interface', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-interface'], 'id_perms': {'created': '2025-03-31T02:15:27.672455', 'creator': None, 'description': 'Vrouter interface(s) down.', 'enable': True, 'last_modified': '2025-03-31T02:15:27.672455', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9979357367960916066, 'uuid_mslong': 750266427534232401}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0a697b57-3c14-4b51-8a7d-cca73bfcd062', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-56816fd6-810a-446e-9778-0684ad64d533', 'type': 'alarm', 'uuid': '0a697b57-3c14-4b51-8a7d-cca73bfcd062'} 03/31/2025 02:15:27.723 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-xmpp-connectivity'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-xmpp-connectivity 03/31/2025 02:15:27.728 7f0b592c0eb8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-connectivity url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-xmpp-connectivity"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["control-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "==", "operand2": {"json_value": "null"}}]}, {"and_list": [{"operand1": "BgpRouterState.num_up_xmpp_peer", "operation": "!=", "operand2": {"uve_attribute": "BgpRouterState.num_xmpp_peer"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "XMPP peer mismatch.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-xmpp-connectivity"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:27.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:27.755 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'json_value': 'null'}, 'operation': '=='}]}, {'and_list': [{'operand1': 'BgpRouterState.num_up_xmpp_peer', 'operand2': {'uve_attribute': 'BgpRouterState.num_xmpp_peer'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-connectivity', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-connectivity'], 'id_perms': {'created': '2025-03-31T02:15:27.742129', 'creator': None, 'description': 'XMPP peer mismatch.', 'enable': True, 'last_modified': '2025-03-31T02:15:27.742129', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13288304350115356995, 'uuid_mslong': 254475288057301197}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '038813ed-1cc5-48cd-b869-8a66e7e76143', 'uve_keys': {'uve_key': ['control-node']}}, 'oper': 'CREATE', 'request-id': 'req-400dde09-0084-49e1-8b95-5b025f0e4792', 'type': 'alarm', 'uuid': '038813ed-1cc5-48cd-b869-8a66e7e76143'} 03/31/2025 02:15:27.805 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-xmpp-close-reason'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-xmpp-close-reason 03/31/2025 02:15:27.810 7f0b59403c18 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-xmpp-close-reason url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-xmpp-close-reason"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["xmpp-peer"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "XmppPeerInfoData.close_reason", "operation": "!=", "operand2": {"json_value": "null"}}, {"operand1": "XmppPeerInfoData.state_info.state", "operation": "!=", "operand2": {"json_value": "\"Established\""}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "XMPP connection closed towards peer, alarm has close reason", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-xmpp-close-reason"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:27.834 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'XmppPeerInfoData.close_reason', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'XmppPeerInfoData.state_info.state', 'operand2': {'json_value': '"Established"'}, 'operation': '!='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-xmpp-close-reason', 'fq_name': ['default-global-system-config', 'system-defined-xmpp-close-reason'], 'id_perms': {'created': '2025-03-31T02:15:27.823156', 'creator': None, 'description': 'XMPP connection closed towards ' 'peer, ' 'alarm has close reason', 'enable': True, 'last_modified': '2025-03-31T02:15:27.823156', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10501937140409570732, 'uuid_mslong': 16038550149706957463}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'de946062-a192-4297-91be-6030039815ac', 'uve_keys': {'uve_key': ['xmpp-peer']}}, 'oper': 'CREATE', 'request-id': 'req-b62d15d0-bec5-49c2-b4f9-3adb6131567f', 'type': 'alarm', 'uuid': 'de946062-a192-4297-91be-6030039815ac'} 03/31/2025 02:15:27.880 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-core-files'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-core-files 03/31/2025 02:15:27.884 7f0b5713b8d0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-core-files url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-core-files"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "null"}, "operation": "!="}, {"operand1": "NodeStatus.all_core_file_list", "operand2": {"json_value": "0"}, "operation": "size!="}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "A core file has been generated on the node.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-core-files"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:27.910 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': 'null'}, 'operation': '!='}, {'operand1': 'NodeStatus.all_core_file_list', 'operand2': {'json_value': '0'}, 'operation': 'size!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-core-files', 'fq_name': ['default-global-system-config', 'system-defined-core-files'], 'id_perms': {'created': '2025-03-31T02:15:27.895662', 'creator': None, 'description': 'A core file has been generated on ' 'the node.', 'enable': True, 'last_modified': '2025-03-31T02:15:27.895662', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13783660346839626274, 'uuid_mslong': 18074982527717559640}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'fad73cd7-f5ff-4d58-bf49-660e76037222', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-f374e460-901c-48bd-a469-2490fabc0cd7', 'type': 'alarm', 'uuid': 'fad73cd7-f5ff-4d58-bf49-660e76037222'} 03/31/2025 02:15:27.947 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-pending-cassandra-compaction-tasks 03/31/2025 02:15:27.954 7f0b59403240 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-pending-cassandra-compaction-tasks url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-pending-cassandra-compaction-tasks"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["database-nodeconfig-database-node"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks", "operand2": {"json_value": "300"}, "operation": ">="}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Pending compaction tasks in cassandra crossed the configured threshold.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-pending-cassandra-compaction-tasks"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:27.977 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'CassandraStatusData.cassandra_compaction_task.pending_compaction_tasks', 'operand2': {'json_value': '300'}, 'operation': '>='}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-pending-cassandra-compaction-tasks', 'fq_name': ['default-global-system-config', 'system-defined-pending-cassandra-compaction-tasks'], 'id_perms': {'created': '2025-03-31T02:15:27.968146', 'creator': None, 'description': 'Pending compaction tasks in ' 'cassandra crossed the configured ' 'threshold.', 'enable': True, 'last_modified': '2025-03-31T02:15:27.968146', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 11815623903089319280, 'uuid_mslong': 902770129283139095}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '0c8748a6-45c1-4617-a3f9-87568740dd70', 'uve_keys': {'uve_key': ['database-nodeconfig-database-node']}}, 'oper': 'CREATE', 'request-id': 'req-4f856ed2-5bf5-4371-a1a6-d1cf39a7b995', 'type': 'alarm', 'uuid': '0c8748a6-45c1-4617-a3f9-87568740dd70'} 03/31/2025 02:15:28.020 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-package-version-mismatch'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-package-version-mismatch 03/31/2025 02:15:28.024 7f0b593228d0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-package-version-mismatch url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-package-version-mismatch"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["analytics-node", "analytics-alarm-node", "analytics-snmp-node", "config-node", "control-node", "database-node", "config-database-node", "vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "NodeStatus.running_package_version", "operation": "!=", "operand2": {"uve_attribute": "NodeStatus.installed_package_version"}}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "There is a mismatch between installed and running package version.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-package-version-mismatch"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:28.047 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'NodeStatus.running_package_version', 'operand2': {'uve_attribute': 'NodeStatus.installed_package_version'}, 'operation': '!='}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-package-version-mismatch', 'fq_name': ['default-global-system-config', 'system-defined-package-version-mismatch'], 'id_perms': {'created': '2025-03-31T02:15:28.036831', 'creator': None, 'description': 'There is a mismatch between ' 'installed and running package ' 'version.', 'enable': True, 'last_modified': '2025-03-31T02:15:28.036831', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 13679422294888847225, 'uuid_mslong': 8200967845229383144}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '71cfb0cd-fd4c-4de8-bdd7-1219bb689779', 'uve_keys': {'uve_key': ['analytics-node', 'analytics-alarm-node', 'analytics-snmp-node', 'config-node', 'control-node', 'database-node', 'config-database-node', 'vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-c525b4b8-5063-46b4-8252-8c5635071bb8', 'type': 'alarm', 'uuid': '71cfb0cd-fd4c-4de8-bdd7-1219bb689779'} 03/31/2025 02:15:28.091 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-limit-exceeded 03/31/2025 02:15:28.095 7f0b592a7438 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-limit-exceeded url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 1, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Agent resource usage exceeded configured watermark for resource.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-limit-exceeded"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:28.114 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 1, 'display_name': 'system-defined-vrouter-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-limit-exceeded'], 'id_perms': {'created': '2025-03-31T02:15:28.106253', 'creator': None, 'description': 'Agent resource usage exceeded ' 'configured watermark for resource.', 'enable': True, 'last_modified': '2025-03-31T02:15:28.106253', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12772578576552700813, 'uuid_mslong': 8424932604656633263}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '74eb5f91-152a-45af-b141-508b19302b8d', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-a4514213-7e4e-4ce8-9a7e-1fbbe96dfdf6', 'type': 'alarm', 'uuid': '74eb5f91-152a-45af-b141-508b19302b8d'} 03/31/2025 02:15:28.161 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-table-limit-exceeded 03/31/2025 02:15:28.168 7f0b59322390 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-table-limit-exceeded url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-table-limit-exceeded"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.res_table_limit", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_limit_exceeded_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "Agent resource usage exceeded table size for resource in vrouter.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-table-limit-exceeded"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:28.203 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.res_table_limit', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_limit_exceeded_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-table-limit-exceeded', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-table-limit-exceeded'], 'id_perms': {'created': '2025-03-31T02:15:28.187405', 'creator': None, 'description': 'Agent resource usage exceeded table ' 'size for resource in vrouter.', 'enable': True, 'last_modified': '2025-03-31T02:15:28.187405', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12425535060279454961, 'uuid_mslong': 11087896127483367341}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '99e01ec8-2113-4fad-ac70-5e4dd41e6cf1', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-fc11b6f6-4fbe-400a-9c32-0c67311fa1fb', 'type': 'alarm', 'uuid': '99e01ec8-2113-4fad-ac70-5e4dd41e6cf1'} 03/31/2025 02:15:28.244 7f0b59322f60 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'] not found in DB: Unknown id: alarm default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down 03/31/2025 02:15:28.249 7f0b5934d4e0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = alarm identifier_name = default-global-system-config:system-defined-vrouter-l3mh-physical-interface-link-down url = https://10.0.0.17:8082/alarms operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_alarm.py remote_ip = 10.0.0.17 body = {"alarm": {"uuid": null, "fq_name": ["default-global-system-config", "system-defined-vrouter-l3mh-physical-interface-link-down"], "parent_type": "global-system-config", "uve_keys": {"uve_key": ["vrouter"]}, "alarm_severity": 0, "alarm_rules": {"or_list": [{"and_list": [{"operand1": "VrouterAgent.l3mh_phy_interface_down", "operation": "==", "operand2": {"json_value": "true"}, "variables": ["VrouterAgent.vr_l3mh_intf_list"]}]}]}, "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": "On l3 multihomed compute one or more physical interfaces are Inactive.", "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "system-defined-vrouter-l3mh-physical-interface-link-down"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:28.274 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'obj_dict': {'alarm_rules': {'or_list': [{'and_list': [{'operand1': 'VrouterAgent.l3mh_phy_interface_down', 'operand2': {'json_value': 'true'}, 'operation': '==', 'variables': ['VrouterAgent.vr_l3mh_intf_list']}]}]}, 'alarm_severity': 0, 'display_name': 'system-defined-vrouter-l3mh-physical-interface-link-down', 'fq_name': ['default-global-system-config', 'system-defined-vrouter-l3mh-physical-interface-link-down'], 'id_perms': {'created': '2025-03-31T02:15:28.261736', 'creator': None, 'description': 'On l3 multihomed compute one or ' 'more physical interfaces are ' 'Inactive.', 'enable': True, 'last_modified': '2025-03-31T02:15:28.261736', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 10625340166528126383, 'uuid_mslong': 14539854520089920173}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'c9c7f0a6-8bd1-42ad-9374-ca9603994daf', 'uve_keys': {'uve_key': ['vrouter']}}, 'oper': 'CREATE', 'request-id': 'req-e4256100-e4c6-4d45-bf7d-463e5986284a', 'type': 'alarm', 'uuid': 'c9c7f0a6-8bd1-42ad-9374-ca9603994daf'} 03/31/2025 02:15:28.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:29.144 7f0b595a4e10 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 1e3d2dbd-97fa-4e55-8bd1-1e895383b50a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = https://10.0.0.17:8082/global-vrouter-config/1e3d2dbd-97fa-4e55-8bd1-1e895383b50a operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_encap.py remote_ip = 10.0.0.17 body = {"global-vrouter-config": {"uuid": "1e3d2dbd-97fa-4e55-8bd1-1e895383b50a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "encapsulation_priorities": {"encapsulation": ["MPLSoUDP", "MPLSoGRE", "VXLAN"]}, "vxlan_network_identifier_mode": "automatic"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:29.154 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-5aabc5af-44df-4511-b52a-2431c5954fc9', 'type': 'global_vrouter_config', 'uuid': '1e3d2dbd-97fa-4e55-8bd1-1e895383b50a'} 03/31/2025 02:15:29.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:29.923 7f0b59405cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_objects in resource global-vrouter-config: 'security_logging_objects' 03/31/2025 02:15:29.924 7f0b59405cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-vrouter-config: 'tag_refs' 03/31/2025 02:15:29.924 7f0b59405cc0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field application_policy_set_back_refs in resource global-vrouter-config: 'application_policy_set_back_refs' 03/31/2025 02:15:30.022 7f0b59322198 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 1e3d2dbd-97fa-4e55-8bd1-1e895383b50a object_type = global_vrouter_config identifier_name = default-global-system-config:default-global-vrouter-config url = https://10.0.0.17:8082/global-vrouter-config/1e3d2dbd-97fa-4e55-8bd1-1e895383b50a operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_global_vrouter_config.py remote_ip = 10.0.0.17 body = {"global-vrouter-config": {"uuid": "1e3d2dbd-97fa-4e55-8bd1-1e895383b50a", "fq_name": ["default-global-system-config", "default-global-vrouter-config"], "parent_type": "global-system-config", "port_translation_pools": {"port_translation_pool": []}}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:30.044 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'default-global-vrouter-config'], 'oper': 'UPDATE', 'request-id': 'req-53a12ede-a904-496d-8325-1bbf482bd632', 'type': 'global_vrouter_config', 'uuid': '1e3d2dbd-97fa-4e55-8bd1-1e895383b50a'} 03/31/2025 02:15:30.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:30.917 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:15:30.917 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:15:30.917 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:15:30.917 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:15:30.917 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:15:30.918 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:15:30.918 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:15:30.918 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:15:30.918 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:15:30.918 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:15:30.918 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:15:30.918 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:15:30.918 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:15:30.918 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:15:30.918 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:15:30.919 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:15:30.919 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:15:30.919 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:15:30.919 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:15:30.919 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:15:30.919 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:15:30.919 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:15:30.919 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:15:30.919 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:15:30.919 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:15:30.920 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:15:30.920 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:15:30.920 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:15:30.920 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:15:30.920 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:15:30.920 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:15:30.920 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:15:30.921 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:15:30.921 7f0b592c1630 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:15:30.974 7f0b595d18d0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 0d5441e0-0a93-4574-96f2-d4e0e141b4e1 object_type = global_system_config identifier_name = default-global-system-config url = https://10.0.0.17:8082/global-system-config/0d5441e0-0a93-4574-96f2-d4e0e141b4e1 operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.17 body = {"global-system-config": {"uuid": "0d5441e0-0a93-4574-96f2-d4e0e141b4e1", "autonomous_system": 64512, "ibgp_auto_mesh": true}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:31.006 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-eea600bf-3d82-46bd-acb9-0e6945c68ebe', 'type': 'global_system_config', 'uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1'} 03/31/2025 02:15:31.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:32.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:33.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:34.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:35.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:36.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:37.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:38.594 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:15:38.595 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:15:38.595 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:15:38.595 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:15:38.595 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:15:38.596 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:15:38.597 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:15:38.597 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:15:38.597 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:15:38.597 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:15:38.597 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:15:38.599 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:15:38.599 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:15:38.599 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:15:38.599 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:15:38.600 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:15:38.605 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:15:38.605 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:15:38.605 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:15:38.605 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:15:38.605 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:15:38.606 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:15:38.606 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:15:38.606 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:15:38.606 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:15:38.606 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:15:38.606 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:15:38.606 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:15:38.607 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:15:38.618 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:15:38.618 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:15:38.618 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:15:38.618 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:15:38.620 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:15:38.732 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 03/31/2025 02:15:38.732 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 03/31/2025 02:15:38.732 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 03/31/2025 02:15:38.732 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 03/31/2025 02:15:38.733 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 03/31/2025 02:15:38.733 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 03/31/2025 02:15:38.733 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 03/31/2025 02:15:38.733 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 03/31/2025 02:15:38.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:38.797 7f0b5925a780 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 0d5441e0-0a93-4574-96f2-d4e0e141b4e1 object_type = global_system_config identifier_name = default-global-system-config url = https://10.0.0.17:8082/global-system-config/0d5441e0-0a93-4574-96f2-d4e0e141b4e1 operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.17 body = {"global-system-config": {"uuid": "0d5441e0-0a93-4574-96f2-d4e0e141b4e1", "ip_fabric_subnets": {"subnet": [{"ip_prefix": "10.0.0.0", "ip_prefix_len": 24}]}}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:38.824 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config'], 'oper': 'UPDATE', 'request-id': 'req-75bbbddd-acba-480a-86c0-cf70b4d64e18', 'type': 'global_system_config', 'uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1'} 03/31/2025 02:15:38.926 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'] not found in DB: Unknown id: virtual_router default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 03/31/2025 02:15:38.930 7f0b5a8486d8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = virtual_router identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 url = https://10.0.0.17:8082/virtual-routers operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.17 body = {"virtual-router": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-juju-os-494-1"], "parent_type": "global-system-config", "virtual_router_dpdk_enabled": false, "virtual_router_ip_address": "10.0.0.17", "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "cn-jenkins-deploy-platform-juju-os-494-1"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:39.016 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'obj_dict': {'display_name': 'cn-jenkins-deploy-platform-juju-os-494-1', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'id_perms': {'created': '2025-03-31T02:15:38.984178', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:38.984178', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 12476957960846986773, 'uuid_mslong': 4242172616174751263}}, 'parent_type': 'global-system-config', 'parent_uuid': '0d5441e0-0a93-4574-96f2-d4e0e141b4e1', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': '3adf3984-b6fc-4e1f-ad27-0f28ddc44e15', 'virtual_router_dpdk_enabled': False, 'virtual_router_ip_address': '10.0.0.17'}, 'oper': 'CREATE', 'request-id': 'req-5bc28261-ee32-4e38-ab85-f15f388e9e72', 'type': 'virtual_router', 'uuid': '3adf3984-b6fc-4e1f-ad27-0f28ddc44e15'} 03/31/2025 02:15:39.018 7f0b5a7f2cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectVRouter:ContrailConfigTrace] map 03/31/2025 02:15:39.112 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:15:39.112 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:15:39.112 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:15:39.112 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:15:39.112 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:15:39.112 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:15:39.112 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:15:39.113 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:15:39.309 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field access_control_lists in resource virtual-network: 'access_control_lists' 03/31/2025 02:15:39.309 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field floating_ip_pools in resource virtual-network: 'floating_ip_pools' 03/31/2025 02:15:39.309 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alias_ip_pools in resource virtual-network: 'alias_ip_pools' 03/31/2025 02:15:39.309 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instances in resource virtual-network: 'routing_instances' 03/31/2025 02:15:39.309 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bridge_domains in resource virtual-network: 'bridge_domains' 03/31/2025 02:15:39.310 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_object_refs in resource virtual-network: 'security_logging_object_refs' 03/31/2025 02:15:39.310 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_refs in resource virtual-network: 'qos_config_refs' 03/31/2025 02:15:39.310 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-network: 'network_ipam_refs' 03/31/2025 02:15:39.310 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_policy_refs in resource virtual-network: 'network_policy_refs' 03/31/2025 02:15:39.310 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_refs in resource virtual-network: 'virtual_network_refs' 03/31/2025 02:15:39.310 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_health_check_refs in resource virtual-network: 'service_health_check_refs' 03/31/2025 02:15:39.310 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_table_refs in resource virtual-network: 'route_table_refs' 03/31/2025 02:15:39.310 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field multicast_policy_refs in resource virtual-network: 'multicast_policy_refs' 03/31/2025 02:15:39.310 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgpvpn_refs in resource virtual-network: 'bgpvpn_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_map_refs in resource virtual-network: 'intent_map_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_refs in resource virtual-network: 'routing_policy_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-network: 'tag_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_network_back_refs in resource virtual-network: 'virtual_network_back_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource virtual-network: 'virtual_machine_interface_back_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-network: 'instance_ip_back_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-network: 'physical_router_back_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field port_tuple_back_refs in resource virtual-network: 'port_tuple_back_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field logical_router_back_refs in resource virtual-network: 'logical_router_back_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_node_back_refs in resource virtual-network: 'flow_node_back_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field firewall_rule_back_refs in resource virtual-network: 'firewall_rule_back_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnect_back_refs in resource virtual-network: 'data_center_interconnect_back_refs' 03/31/2025 02:15:39.311 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabric_back_refs in resource virtual-network: 'fabric_back_refs' 03/31/2025 02:15:39.312 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field host_based_service_back_refs in resource virtual-network: 'host_based_service_back_refs' 03/31/2025 02:15:39.383 7f0b595d1978 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Object with fq_name ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1', 'vhost0'] not found in DB: Unknown id: virtual_machine_interface default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1:vhost0 03/31/2025 02:15:39.401 7f0b595a1978 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << object_type = virtual_machine_interface identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1:vhost0 url = https://10.0.0.17:8082/virtual-machine-interfaces operation = post useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.17 body = {"virtual-machine-interface": {"uuid": null, "fq_name": ["default-global-system-config", "cn-jenkins-deploy-platform-juju-os-494-1", "vhost0"], "parent_type": "virtual-router", "virtual_machine_interface_disable_policy": true, "virtual_network_refs": [{"to": ["default-domain", "default-project", "ip-fabric"], "uuid": "3da78d9c-3d98-4934-a990-647a4cdbfe7f"}], "id_perms": {"permissions": {"owner": "cloud-admin", "owner_access": 7, "group": "cloud-admin-group", "group_access": 7, "other_access": 7}, "uuid": null, "enable": true, "created": null, "last_modified": null, "description": null, "user_visible": true, "creator": null}, "perms2": {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}, "display_name": "vhost0"}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:39.522 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1', 'vhost0'], 'obj_dict': {'display_name': 'vhost0', 'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1', 'vhost0'], 'id_perms': {'created': '2025-03-31T02:15:39.477682', 'creator': None, 'description': None, 'enable': True, 'last_modified': '2025-03-31T02:15:39.477682', 'permissions': {'group': 'admin', 'group_access': 7, 'other_access': 7, 'owner': 'admin', 'owner_access': 7}, 'user_visible': True, 'uuid': {'uuid_lslong': 9410503086102487133, 'uuid_mslong': 14746711583981650961}}, 'parent_type': 'virtual-router', 'parent_uuid': '3adf3984-b6fc-4e1f-ad27-0f28ddc44e15', 'perms2': {'global_access': 0, 'owner': 'cloud-admin', 'owner_access': 7, 'share': []}, 'uuid': 'cca6d80d-e05d-4411-8298-d2b2a4db205d', 'virtual_machine_interface_disable_policy': True, 'virtual_machine_interface_mac_addresses': {'mac_address': ['02:cc:a6:d8:0d:e0']}, 'virtual_network_refs': [{'to': ['default-domain', 'default-project', 'ip-fabric'], 'uuid': '3da78d9c-3d98-4934-a990-647a4cdbfe7f'}]}, 'oper': 'CREATE', 'request-id': 'req-c4c69a9a-97bf-45e1-bb95-960998a7cb34', 'type': 'virtual_machine_interface', 'uuid': 'cca6d80d-e05d-4411-8298-d2b2a4db205d'} 03/31/2025 02:15:39.522 7f0b5a7f2cc0 [contrail-api] [DEBUG]: Add uve in the [ObjectVMITable:ContrailConfigTrace] map 03/31/2025 02:15:39.590 7f0b5925a6d8 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = cca6d80d-e05d-4411-8298-d2b2a4db205d object_type = virtual_machine_interface identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1:vhost0 url = http://127.0.0.1/ref-update operation = ref-update domain = default-domain >> 03/31/2025 02:15:39.593 7f0b595b2cc0 [contrail-api] [INFO]: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 57fa88d5-1162-43c2-ba82-a5761f77fd36 object_type = access_control_list identifier_name = default-domain:default-project:ip-fabric:ip-fabric url = https://10.0.0.17:8082/access-control-list/57fa88d5-1162-43c2-ba82-a5761f77fd36 operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/usr/local/bin/contrail-schema remote_ip = 10.0.0.17 body = {"access-control-list": {"uuid": "57fa88d5-1162-43c2-ba82-a5761f77fd36", "access_control_list_entries": {"dynamic": false, "acl_rule": [{"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "default-domain:default-project:ip-fabric", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "deny", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000002", "direction": "<>"}, {"match_condition": {"protocol": "any", "src_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "src_port": {"start_port": 0, "end_port": 65535}, "dst_address": {"subnet": null, "virtual_network": "any", "security_group": null, "network_policy": null, "subnet_list": []}, "dst_port": {"start_port": 0, "end_port": 65535}, "ethertype": null}, "action_list": {"simple_action": "pass", "gateway_name": null, "apply_service": [], "service_properties": null, "mirror_to": null, "assign_routing_instance": null, "log": false, "alert": false, "qos_action": null, "host_based_service": false}, "rule_uuid": "00000000-0000-0000-0000-100000000001", "direction": null}]}, "access_control_list_hash": -7096420548783179476}} domain = default-domain project = admin user = admin >> 03/31/2025 02:15:39.665 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-c4c69a9a-97bf-45e1-bb95-960998a7cb34', 'type': 'virtual_machine_interface', 'uuid': 'cca6d80d-e05d-4411-8298-d2b2a4db205d'} 03/31/2025 02:15:39.688 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', 'ip-fabric'], 'oper': 'UPDATE', 'request-id': 'req-7ce1b7ac-0b06-40c1-87fb-211724985d9f', 'type': 'access_control_list', 'uuid': '57fa88d5-1162-43c2-ba82-a5761f77fd36'} 03/31/2025 02:15:39.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:40.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:41.547 7f0b59ab7e10 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:15:41.613 7f0b59ab7e10 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:15:41.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:42.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:43.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:44.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:46.712 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:47.719 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:48.720 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:49.722 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:50.722 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:51.723 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:52.657 7f0b5a7f2198 [contrail-api] [ERROR]: Session Event: TCP Connection Closed 03/31/2025 02:15:52.657 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvTcpClose] in state[Established] 03/31/2025 02:15:52.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:53.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:54.728 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:55.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:56.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:57.659 7f0b59ab7e10 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:15:57.659 7f0b59ab7e10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpClose] => State[Established] -> State[Connect] 03/31/2025 02:15:57.660 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:15:57.660 7f0b59ab7e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Established to Connect on EvTcpClose >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Established to Connect on EvTcpClose] connection down >>, ] >> 03/31/2025 02:15:57.660 7f0b59ab7e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:15:57.660 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:15:57.661 7f0b59ab7e10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:15:57.661 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:15:57.661 7f0b59ab7e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:15:57.661 7f0b59ab7e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:15:57.661 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:15:57.661 7f0b59ab7e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:15:57.661 7f0b59ab7e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:15:57.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:58.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:15:59.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:00.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:01.661 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:16:01.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:02.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:02.846 7f0b595b2eb8 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:16:02.897 7f0b595b2eb8 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:16:03.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:04.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:05.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:06.664 7f0b59ab7e10 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:16:06.665 7f0b59ab7e10 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:16:06.665 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:06.665 7f0b59ab7e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:16:06.665 7f0b59ab7e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:06.665 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:16:06.665 7f0b59ab7e10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:16:06.666 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:06.666 7f0b59ab7e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:16:06.666 7f0b59ab7e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:06.666 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:06.666 7f0b59ab7e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:16:06.666 7f0b59ab7e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:06.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:07.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:08.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:09.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:10.666 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:16:10.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:11.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:12.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:13.439 7f0b592d7cc0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:16:13.490 7f0b592d7cc0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:16:13.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:14.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:15.671 7f0b59ab7e10 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:16:15.671 7f0b59ab7e10 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:16:15.671 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:15.672 7f0b59ab7e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:16:15.672 7f0b59ab7e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:15.672 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:16:15.673 7f0b59ab7e10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:16:15.682 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:15.683 7f0b59ab7e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:16:15.683 7f0b59ab7e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:15.683 7f0b59ab7e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:15.683 7f0b59ab7e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:16:15.683 7f0b59ab7e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:16.713 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:17.715 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:18.716 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:19.671 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:16:19.718 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:20.720 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:20.909 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:16:20.910 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:16:20.910 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:16:20.910 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:16:20.910 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:16:20.910 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:16:20.910 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:16:20.910 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:16:21.722 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:22.723 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:23.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:24.675 7f0b5a8080f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:16:24.675 7f0b5a8080f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:16:24.675 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:24.675 7f0b5a8080f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:16:24.675 7f0b5a8080f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:24.675 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:16:24.676 7f0b5a8080f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:16:24.676 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:24.676 7f0b5a8080f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:16:24.676 7f0b5a8080f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:24.676 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:24.676 7f0b5a8080f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:16:24.676 7f0b5a8080f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:24.726 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:25.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:26.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:27.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:28.676 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:16:28.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:29.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:30.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:31.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:32.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:33.680 7f0b5a8080f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:16:33.680 7f0b5a8080f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:16:33.680 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:33.680 7f0b5a8080f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:16:33.680 7f0b5a8080f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:33.680 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:16:33.681 7f0b5a8080f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:16:33.681 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:33.681 7f0b5a8080f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:16:33.681 7f0b5a8080f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:33.681 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:33.681 7f0b5a8080f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:16:33.681 7f0b5a8080f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:33.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:34.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:35.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:36.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:37.681 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:16:37.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:38.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:39.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:40.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:41.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:42.628 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:16:42.628 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:16:42.628 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:16:42.628 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:16:42.628 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:16:42.628 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:16:42.629 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:16:42.629 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:16:42.629 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:16:42.629 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:16:42.629 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:16:42.629 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:16:42.629 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:16:42.629 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:16:42.629 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:16:42.629 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:16:42.629 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:16:42.630 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:16:42.630 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:16:42.630 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:16:42.630 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:16:42.630 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:16:42.630 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:16:42.630 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:16:42.630 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:16:42.630 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:16:42.630 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:16:42.630 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:16:42.631 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:16:42.631 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:16:42.631 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:16:42.631 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:16:42.631 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:16:42.631 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:16:42.683 7f0b5a8080f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:16:42.683 7f0b5a8080f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:16:42.683 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:42.684 7f0b5a8080f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:16:42.684 7f0b5a8080f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:42.684 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:16:42.684 7f0b5a8080f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:16:42.684 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:42.685 7f0b5a8080f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:16:42.685 7f0b5a8080f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:42.685 7f0b5a8080f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:42.685 7f0b5a8080f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:16:42.685 7f0b5a8080f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:42.733 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource config-node: 'tag_refs' 03/31/2025 02:16:42.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:43.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:44.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:46.684 7f0b5998de10 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:16:46.713 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:47.715 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:48.716 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:49.717 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:50.718 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:51.688 7f0b5998de10 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:16:51.688 7f0b5998de10 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:16:51.688 7f0b5998de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:51.689 7f0b5998de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:16:51.689 7f0b5998de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:16:51.689 7f0b5998de10 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:16:51.689 7f0b5998de10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:16:51.689 7f0b5998de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:51.689 7f0b5998de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:16:51.689 7f0b5998de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:51.689 7f0b5998de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:51.690 7f0b5998de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:16:51.690 7f0b5998de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:51.719 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:52.721 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:53.722 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:54.531 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 03/31/2025 02:16:54.531 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 03/31/2025 02:16:54.531 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 03/31/2025 02:16:54.532 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 03/31/2025 02:16:54.532 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 03/31/2025 02:16:54.532 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 03/31/2025 02:16:54.532 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 03/31/2025 02:16:54.532 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 03/31/2025 02:16:54.678 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zone_refs in resource bgp-router: 'control_node_zone_refs' 03/31/2025 02:16:54.678 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource bgp-router: 'sub_cluster_refs' 03/31/2025 02:16:54.678 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource bgp-router: 'bgp_router_refs' 03/31/2025 02:16:54.678 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource bgp-router: 'tag_refs' 03/31/2025 02:16:54.678 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource bgp-router: 'physical_router_back_refs' 03/31/2025 02:16:54.678 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource bgp-router: 'virtual_machine_interface_back_refs' 03/31/2025 02:16:54.679 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_as_a_service_back_refs in resource bgp-router: 'bgp_as_a_service_back_refs' 03/31/2025 02:16:54.679 7f0b5998de10 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_back_refs in resource bgp-router: 'bgp_router_back_refs' 03/31/2025 02:16:54.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:54.728 7f0b5998de10 [contrail-api] [ERROR]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = a4cca4e6-ac05-4bd1-8b63-e44a647ea6c6 object_type = bgp_router identifier_name = default-domain:default-project:ip-fabric:__default__:cn-jenkins-deploy-platform-juju-os-494-1 url = https://10.0.0.17:8082/bgp-router/a4cca4e6-ac05-4bd1-8b63-e44a647ea6c6 operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_control.py remote_ip = 10.0.0.17 body = {"bgp-router": {"uuid": "a4cca4e6-ac05-4bd1-8b63-e44a647ea6c6", "bgp_router_parameters": {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.0.0.17", "address": "10.0.0.17", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}}} domain = default-domain project = admin user = admin >> 03/31/2025 02:16:54.742 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-domain', 'default-project', 'ip-fabric', '__default__', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'oper': 'UPDATE', 'request-id': 'req-673470b0-8fd7-40f5-8570-0a4ffca4a58a', 'type': 'bgp_router', 'uuid': 'a4cca4e6-ac05-4bd1-8b63-e44a647ea6c6'} 03/31/2025 02:16:54.746 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:54.746 7f0b592608d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 elements = { bgp_router_parameters : {"admin_down": false, "vendor": "contrail", "cluster_id": null, "autonomous_system": 64512, "identifier": "10.0.0.17", "address": "10.0.0.17", "port": 179, "source_port": null, "hold_time": 0, "address_families": {"family": ["route-target", "inet-vpn", "e-vpn", "erm-vpn", "inet6-vpn"]}, "auth_data": null, "local_autonomous_system": null, "router_type": "control-node", "gateway_address": null, "ipv6_gateway_address": null}display_name : "cn-jenkins-deploy-platform-juju-os-494-1"fq_name : ["default-domain", "default-project", "ip-fabric", "__default__", "cn-jenkins-deploy-platform-juju-os-494-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 11875047628104616913, "uuid_lslong": 10044122602131072710}, "enable": true, "created": "2025-03-31T02:15:23.817579", "last_modified": "2025-03-31T02:16:54.736519", "description": null, "user_visible": true, "creator": null}parent_type : "routing-instance"parent_uuid : "d7ea1fdf-c27f-4008-8f8f-5741e27e1e75"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "a4cca4e6-ac05-4bd1-8b63-e44a647ea6c6" } deleted = False >> 03/31/2025 02:16:54.746 7f0b592608d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:16:54.976 7f0b592608d0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:16:55.029 7f0b592608d0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:16:55.689 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:16:55.726 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:56.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:57.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:58.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:16:59.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:00.692 7f0b592608d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:17:00.692 7f0b592608d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:17:00.693 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:00.693 7f0b592608d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:17:00.693 7f0b592608d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:00.693 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:17:00.693 7f0b592608d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:17:00.693 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:00.694 7f0b592608d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:17:00.694 7f0b592608d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:00.694 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:00.694 7f0b592608d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:17:00.694 7f0b592608d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:00.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:01.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:02.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:03.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:04.693 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:17:04.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:05.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:06.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:07.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:08.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:09.695 7f0b592608d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:17:09.695 7f0b592608d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:17:09.695 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:09.696 7f0b592608d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:17:09.696 7f0b592608d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:09.696 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:17:09.696 7f0b592608d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:17:09.696 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:09.696 7f0b592608d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:17:09.697 7f0b592608d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:09.697 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:09.697 7f0b592608d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:17:09.697 7f0b592608d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:09.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:10.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:11.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:12.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:13.696 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:17:13.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:14.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:16.715 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:17.716 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:18.700 7f0b592608d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:17:18.700 7f0b592608d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:17:18.700 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:18.700 7f0b592608d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:17:18.701 7f0b592608d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:18.701 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:17:18.701 7f0b592608d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:17:18.701 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:18.701 7f0b592608d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:17:18.701 7f0b592608d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:18.701 7f0b592608d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:18.702 7f0b592608d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:17:18.702 7f0b592608d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:18.717 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:19.719 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:20.721 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:21.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:21.813 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:17:21.813 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:17:21.813 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:17:21.813 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:17:21.813 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:17:21.813 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:17:21.814 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:17:21.814 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:17:22.701 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:17:22.728 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:23.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:24.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:25.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:26.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:27.705 7f0b59260b70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:17:27.705 7f0b59260b70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:17:27.705 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:27.705 7f0b59260b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:17:27.705 7f0b59260b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:27.706 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:17:27.706 7f0b59260b70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:17:27.706 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:27.706 7f0b59260b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:17:27.706 7f0b59260b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:27.706 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:27.706 7f0b59260b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:17:27.706 7f0b59260b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:27.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:28.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:29.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:30.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:31.706 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:17:31.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:32.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:33.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:34.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:35.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:36.709 7f0b59260b70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:17:36.709 7f0b59260b70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:17:36.709 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:36.709 7f0b59260b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:17:36.710 7f0b59260b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:36.710 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:17:36.710 7f0b59260b70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:17:36.710 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:36.710 7f0b59260b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:17:36.710 7f0b59260b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:36.710 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:36.711 7f0b59260b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:17:36.711 7f0b59260b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:36.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:37.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:38.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:39.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:40.710 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:17:40.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:41.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:42.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:43.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:44.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:45.713 7f0b59260b70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:17:45.713 7f0b59260b70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:17:45.713 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:45.713 7f0b59260b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:17:45.713 7f0b59260b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:45.713 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:17:45.714 7f0b59260b70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:17:45.714 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:45.714 7f0b59260b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:17:45.714 7f0b59260b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:45.714 7f0b59260b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:45.714 7f0b59260b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:17:45.715 7f0b59260b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:46.715 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:47.717 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:48.719 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:49.714 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:17:49.720 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:50.721 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:51.722 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:52.723 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:53.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:54.717 7f0b592c1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:17:54.717 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:17:54.717 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:54.717 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:17:54.717 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:17:54.718 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:17:54.718 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:17:54.718 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:54.718 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:17:54.718 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:54.718 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:54.718 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:17:54.719 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:17:54.726 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:55.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:56.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:57.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:58.718 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:17:58.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:17:59.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:00.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:01.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:02.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:03.721 7f0b664aaf60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:18:03.721 7f0b664aaf60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:18:03.721 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:03.721 7f0b664aaf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:18:03.721 7f0b664aaf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:03.721 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:18:03.722 7f0b664aaf60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:18:03.722 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:03.722 7f0b664aaf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:18:03.722 7f0b664aaf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:03.722 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:03.722 7f0b664aaf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:18:03.722 7f0b664aaf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:03.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:04.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:05.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:06.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:07.722 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:18:07.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:08.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:09.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:10.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:11.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:12.726 7f0b664aaf60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:18:12.726 7f0b664aaf60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:18:12.726 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:12.727 7f0b664aaf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:18:12.727 7f0b664aaf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:12.727 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:18:12.727 7f0b664aaf60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:18:12.727 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:12.728 7f0b664aaf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:18:12.728 7f0b664aaf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:12.728 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:12.728 7f0b664aaf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:18:12.728 7f0b664aaf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:12.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:13.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:14.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:16.716 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:16.726 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:18:17.718 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:18.720 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:19.722 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:20.724 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:21.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:21.729 7f0b664aaf60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:18:21.729 7f0b664aaf60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:18:21.729 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:21.729 7f0b664aaf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:18:21.729 7f0b664aaf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:21.729 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:18:21.730 7f0b664aaf60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:18:21.730 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:21.730 7f0b664aaf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:18:21.730 7f0b664aaf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:21.730 7f0b664aaf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:21.730 7f0b664aaf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:18:21.731 7f0b664aaf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:22.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:22.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:18:22.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:18:22.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:18:22.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:18:22.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:18:22.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:18:22.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:18:22.739 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:18:23.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:24.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:25.730 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:18:25.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:26.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:27.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:28.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:29.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:30.733 7f0b5712ee10 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:18:30.734 7f0b5712ee10 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:18:30.734 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:30.734 7f0b5712ee10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:18:30.734 7f0b5712ee10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:30.734 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:18:30.735 7f0b5712ee10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:18:30.735 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:30.735 7f0b5712ee10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:18:30.735 7f0b5712ee10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:30.735 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:30.735 7f0b5712ee10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:18:30.735 7f0b5712ee10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:30.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:31.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:32.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:33.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:34.735 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:18:34.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:35.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:36.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:37.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:38.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:39.737 7f0b5712ee10 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:18:39.737 7f0b5712ee10 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:18:39.737 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:39.737 7f0b5712ee10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:18:39.737 7f0b5712ee10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:39.737 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:18:39.738 7f0b5712ee10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:18:39.738 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:39.738 7f0b5712ee10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:18:39.738 7f0b5712ee10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:39.738 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:39.738 7f0b5712ee10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:18:39.738 7f0b5712ee10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:39.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:40.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:41.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:42.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:43.738 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:18:43.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:44.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:46.717 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:47.719 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:48.721 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:48.740 7f0b5712ee10 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:18:48.740 7f0b5712ee10 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:18:48.741 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:48.741 7f0b5712ee10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:18:48.741 7f0b5712ee10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:48.741 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:18:48.741 7f0b5712ee10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:18:48.741 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:48.742 7f0b5712ee10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:18:48.742 7f0b5712ee10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:48.742 7f0b5712ee10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:48.742 7f0b5712ee10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:18:48.742 7f0b5712ee10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:49.723 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:50.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:51.726 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:52.728 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:52.741 7f0b59260240 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:18:53.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:54.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:55.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:56.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:57.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:57.743 7f0b59260240 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:18:57.743 7f0b59260240 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:18:57.743 7f0b59260240 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:57.743 7f0b59260240 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:18:57.744 7f0b59260240 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:18:57.744 7f0b59260240 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:18:57.744 7f0b59260240 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:18:57.744 7f0b59260240 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:57.744 7f0b59260240 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:18:57.744 7f0b59260240 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:57.744 7f0b59260240 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:57.745 7f0b59260240 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:18:57.745 7f0b59260240 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:18:58.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:18:59.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:00.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:01.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:01.744 7f0b59260240 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:19:02.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:03.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:04.725 7f0b59405d68 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:19:04.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:04.776 7f0b59405d68 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:19:05.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:06.746 7f0b59260240 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:19:06.747 7f0b59260240 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:19:06.747 7f0b59260240 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:06.747 7f0b59260240 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:19:06.747 7f0b59260240 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:06.747 7f0b59260240 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:19:06.747 7f0b59260240 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:19:06.748 7f0b59260240 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:06.748 7f0b59260240 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:19:06.748 7f0b59260240 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:06.748 7f0b59260240 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:06.748 7f0b59260240 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:19:06.748 7f0b59260240 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:06.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:07.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:08.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:09.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:10.747 7f0b59260240 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:19:10.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:11.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:12.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:13.489 7f0b59405780 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:19:13.540 7f0b59405780 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:19:13.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:14.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:15.749 7f0b59260240 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:19:15.749 7f0b59260240 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:19:15.749 7f0b59260240 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:15.749 7f0b59260240 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:19:15.749 7f0b59260240 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:15.749 7f0b59260240 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:19:15.750 7f0b59260240 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:19:15.750 7f0b59260240 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:15.750 7f0b59260240 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:19:15.750 7f0b59260240 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:15.750 7f0b59260240 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:15.750 7f0b59260240 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:19:15.750 7f0b59260240 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:16.718 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:17.720 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:18.721 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:19.723 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:19.749 7f0b5936cb70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:19:20.724 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:21.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:22.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:23.722 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:19:23.722 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:19:23.722 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:19:23.722 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:19:23.723 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:19:23.723 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:19:23.723 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:19:23.723 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:19:23.728 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:24.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:24.751 7f0b5936cb70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:19:24.751 7f0b5936cb70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:19:24.751 7f0b5936cb70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:24.751 7f0b5936cb70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:19:24.751 7f0b5936cb70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:24.751 7f0b5936cb70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:19:24.752 7f0b5936cb70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:19:24.752 7f0b5936cb70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:24.752 7f0b5936cb70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:19:24.752 7f0b5936cb70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:24.752 7f0b5936cb70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:24.752 7f0b5936cb70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:19:24.752 7f0b5936cb70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:25.093 7f0b5932c4e0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:19:25.145 7f0b5932c4e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:19:25.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:26.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:27.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:28.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:28.751 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:19:29.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:30.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:31.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:32.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:33.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:33.753 7f0b5932c4e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:19:33.753 7f0b5932c4e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:19:33.753 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:33.754 7f0b5932c4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:19:33.754 7f0b5932c4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:33.754 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:19:33.754 7f0b5932c4e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:19:33.754 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:33.754 7f0b5932c4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:19:33.754 7f0b5932c4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:33.754 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:33.754 7f0b5932c4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:19:33.755 7f0b5932c4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:34.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:35.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:36.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:37.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:37.758 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:19:38.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:39.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:40.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:41.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:42.762 7f0b5932c4e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:19:42.762 7f0b5932c4e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:19:42.762 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:42.762 7f0b5932c4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:19:42.762 7f0b5932c4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:42.762 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:19:42.763 7f0b5932c4e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:19:42.763 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:42.763 7f0b5932c4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:19:42.763 7f0b5932c4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:42.763 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:42.763 7f0b5932c4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:19:42.763 7f0b5932c4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:42.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:43.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:44.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:46.722 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:46.763 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:19:47.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:47.980 7f0b5912d828 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:19:48.031 7f0b5912fe10 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:19:48.063 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Parent key not found for resource global-system-config: 'parent_type' 03/31/2025 02:19:48.063 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field feature_flags in resource global-system-config: 'feature_flags' 03/31/2025 02:19:48.064 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field control_node_zones in resource global-system-config: 'control_node_zones' 03/31/2025 02:19:48.064 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_vrouter_configs in resource global-system-config: 'global_vrouter_configs' 03/31/2025 02:19:48.064 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_qos_configs in resource global-system-config: 'global_qos_configs' 03/31/2025 02:19:48.065 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_routers in resource global-system-config: 'virtual_routers' 03/31/2025 02:19:48.065 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_nodes in resource global-system-config: 'config_nodes' 03/31/2025 02:19:48.066 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_nodes in resource global-system-config: 'analytics_nodes' 03/31/2025 02:19:48.066 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field flow_nodes in resource global-system-config: 'flow_nodes' 03/31/2025 02:19:48.066 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field devicemgr_nodes in resource global-system-config: 'devicemgr_nodes' 03/31/2025 02:19:48.067 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field database_nodes in resource global-system-config: 'database_nodes' 03/31/2025 02:19:48.067 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field webui_nodes in resource global-system-config: 'webui_nodes' 03/31/2025 02:19:48.067 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_database_nodes in resource global-system-config: 'config_database_nodes' 03/31/2025 02:19:48.067 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_alarm_nodes in resource global-system-config: 'analytics_alarm_nodes' 03/31/2025 02:19:48.068 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field analytics_snmp_nodes in resource global-system-config: 'analytics_snmp_nodes' 03/31/2025 02:19:48.069 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_appliance_sets in resource global-system-config: 'service_appliance_sets' 03/31/2025 02:19:48.069 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field api_access_lists in resource global-system-config: 'api_access_lists' 03/31/2025 02:19:48.070 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alarms in resource global-system-config: 'alarms' 03/31/2025 02:19:48.070 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field config_propertiess in resource global-system-config: 'config_propertiess' 03/31/2025 02:19:48.070 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field job_templates in resource global-system-config: 'job_templates' 03/31/2025 02:19:48.071 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field data_center_interconnects in resource global-system-config: 'data_center_interconnects' 03/31/2025 02:19:48.071 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field intent_maps in resource global-system-config: 'intent_maps' 03/31/2025 02:19:48.072 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field fabrics in resource global-system-config: 'fabrics' 03/31/2025 02:19:48.072 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field node_profiles in resource global-system-config: 'node_profiles' 03/31/2025 02:19:48.073 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_routers in resource global-system-config: 'physical_routers' 03/31/2025 02:19:48.073 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field device_images in resource global-system-config: 'device_images' 03/31/2025 02:19:48.074 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field nodes in resource global-system-config: 'nodes' 03/31/2025 02:19:48.074 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field features in resource global-system-config: 'features' 03/31/2025 02:19:48.074 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_roles in resource global-system-config: 'physical_roles' 03/31/2025 02:19:48.074 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field overlay_roles in resource global-system-config: 'overlay_roles' 03/31/2025 02:19:48.074 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field role_definitions in resource global-system-config: 'role_definitions' 03/31/2025 02:19:48.074 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field global_analytics_configs in resource global-system-config: 'global_analytics_configs' 03/31/2025 02:19:48.074 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource global-system-config: 'tag_refs' 03/31/2025 02:19:48.074 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_back_refs in resource global-system-config: 'qos_config_back_refs' 03/31/2025 02:19:48.179 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_routers in resource routing-instance: 'bgp_routers' 03/31/2025 02:19:48.179 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_refs in resource routing-instance: 'routing_instance_refs' 03/31/2025 02:19:48.179 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_target_refs in resource routing-instance: 'route_target_refs' 03/31/2025 02:19:48.179 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource routing-instance: 'tag_refs' 03/31/2025 02:19:48.180 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource routing-instance: 'virtual_machine_interface_back_refs' 03/31/2025 02:19:48.180 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field route_aggregate_back_refs in resource routing-instance: 'route_aggregate_back_refs' 03/31/2025 02:19:48.180 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_policy_back_refs in resource routing-instance: 'routing_policy_back_refs' 03/31/2025 02:19:48.180 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field routing_instance_back_refs in resource routing-instance: 'routing_instance_back_refs' 03/31/2025 02:19:48.283 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:19:48.283 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:19:48.283 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:19:48.283 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:19:48.283 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:19:48.283 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:19:48.283 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:19:48.283 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:19:48.333 7f0b59138ac8 [contrail-api] [ERROR]: SANDESH: Not connected: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 3adf3984-b6fc-4e1f-ad27-0f28ddc44e15 object_type = virtual_router identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 url = https://10.0.0.17:8082/virtual-router/3adf3984-b6fc-4e1f-ad27-0f28ddc44e15 operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.17 body = {"virtual-router": {"uuid": "3adf3984-b6fc-4e1f-ad27-0f28ddc44e15", "virtual_router_dpdk_enabled": false}} domain = default-domain project = admin user = admin >> 03/31/2025 02:19:48.333 7f0b59138ac8 [contrail-api] [ERROR]: SANDESH: [DROP: SessionNotConnected] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = 3adf3984-b6fc-4e1f-ad27-0f28ddc44e15 object_type = virtual_router identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1 url = https://10.0.0.17:8082/virtual-router/3adf3984-b6fc-4e1f-ad27-0f28ddc44e15 operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.17 body = {"virtual-router": {"uuid": "3adf3984-b6fc-4e1f-ad27-0f28ddc44e15", "virtual_router_dpdk_enabled": false}} domain = default-domain project = admin user = admin >> 03/31/2025 02:19:48.360 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1'], 'oper': 'UPDATE', 'request-id': 'req-0661aa0c-cdec-4460-9e2e-7bbc4b7c4ab7', 'type': 'virtual_router', 'uuid': '3adf3984-b6fc-4e1f-ad27-0f28ddc44e15'} 03/31/2025 02:19:48.463 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:19:48.463 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:19:48.463 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:19:48.463 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:19:48.463 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:19:48.463 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:19:48.464 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:19:48.464 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:19:48.610 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_logging_object_refs in resource virtual-machine-interface: 'security_logging_object_refs' 03/31/2025 02:19:48.610 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field qos_config_refs in resource virtual-machine-interface: 'qos_config_refs' 03/31/2025 02:19:48.610 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field security_group_refs in resource virtual-machine-interface: 'security_group_refs' 03/31/2025 02:19:48.610 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_refs in resource virtual-machine-interface: 'virtual_machine_interface_refs' 03/31/2025 02:19:48.611 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-machine-interface: 'virtual_machine_refs' 03/31/2025 02:19:48.611 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_router_refs in resource virtual-machine-interface: 'bgp_router_refs' 03/31/2025 02:19:48.611 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field port_tuple_refs in resource virtual-machine-interface: 'port_tuple_refs' 03/31/2025 02:19:48.612 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_health_check_refs in resource virtual-machine-interface: 'service_health_check_refs' 03/31/2025 02:19:48.612 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field interface_route_table_refs in resource virtual-machine-interface: 'interface_route_table_refs' 03/31/2025 02:19:48.612 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_interface_refs in resource virtual-machine-interface: 'physical_interface_refs' 03/31/2025 02:19:48.613 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bridge_domain_refs in resource virtual-machine-interface: 'bridge_domain_refs' 03/31/2025 02:19:48.613 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field service_endpoint_refs in resource virtual-machine-interface: 'service_endpoint_refs' 03/31/2025 02:19:48.613 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_port_group_refs in resource virtual-machine-interface: 'virtual_port_group_refs' 03/31/2025 02:19:48.613 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field port_profile_refs in resource virtual-machine-interface: 'port_profile_refs' 03/31/2025 02:19:48.613 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-machine-interface: 'tag_refs' 03/31/2025 02:19:48.613 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interface_back_refs in resource virtual-machine-interface: 'virtual_machine_interface_back_refs' 03/31/2025 02:19:48.614 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-machine-interface: 'instance_ip_back_refs' 03/31/2025 02:19:48.614 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field subnet_back_refs in resource virtual-machine-interface: 'subnet_back_refs' 03/31/2025 02:19:48.614 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field floating_ip_back_refs in resource virtual-machine-interface: 'floating_ip_back_refs' 03/31/2025 02:19:48.614 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field alias_ip_back_refs in resource virtual-machine-interface: 'alias_ip_back_refs' 03/31/2025 02:19:48.614 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field logical_interface_back_refs in resource virtual-machine-interface: 'logical_interface_back_refs' 03/31/2025 02:19:48.614 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field bgp_as_a_service_back_refs in resource virtual-machine-interface: 'bgp_as_a_service_back_refs' 03/31/2025 02:19:48.614 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field customer_attachment_back_refs in resource virtual-machine-interface: 'customer_attachment_back_refs' 03/31/2025 02:19:48.614 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field logical_router_back_refs in resource virtual-machine-interface: 'logical_router_back_refs' 03/31/2025 02:19:48.616 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field loadbalancer_pool_back_refs in resource virtual-machine-interface: 'loadbalancer_pool_back_refs' 03/31/2025 02:19:48.616 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_ip_back_refs in resource virtual-machine-interface: 'virtual_ip_back_refs' 03/31/2025 02:19:48.616 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field loadbalancer_back_refs in resource virtual-machine-interface: 'loadbalancer_back_refs' 03/31/2025 02:19:48.616 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_port_group_back_refs in resource virtual-machine-interface: 'virtual_port_group_back_refs' 03/31/2025 02:19:48.616 7f0b5936ceb8 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field link_aggregation_group_back_refs in resource virtual-machine-interface: 'link_aggregation_group_back_refs' 03/31/2025 02:19:48.669 7f0b59291630 [contrail-api] [ERROR]: SANDESH: Not connected: __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = cca6d80d-e05d-4411-8298-d2b2a4db205d object_type = virtual_machine_interface identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1:vhost0 url = https://10.0.0.17:8082/virtual-machine-interface/cca6d80d-e05d-4411-8298-d2b2a4db205d operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.17 body = {"virtual-machine-interface": {"uuid": "cca6d80d-e05d-4411-8298-d2b2a4db205d", "virtual_machine_interface_disable_policy": true}} domain = default-domain project = admin user = admin >> 03/31/2025 02:19:48.669 7f0b59291630 [contrail-api] [ERROR]: SANDESH: [DROP: SessionNotConnected] __default__ [SYS_INFO]: VncApiConfigLog: api_log = << identifier_uuid = cca6d80d-e05d-4411-8298-d2b2a4db205d object_type = virtual_machine_interface identifier_name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1:vhost0 url = https://10.0.0.17:8082/virtual-machine-interface/cca6d80d-e05d-4411-8298-d2b2a4db205d operation = http_put useragent = cn-jenkins-deploy-platform-juju-os-494-1:/opt/contrail/utils/provision_vrouter.py remote_ip = 10.0.0.17 body = {"virtual-machine-interface": {"uuid": "cca6d80d-e05d-4411-8298-d2b2a4db205d", "virtual_machine_interface_disable_policy": true}} domain = default-domain project = admin user = admin >> 03/31/2025 02:19:48.694 7f0b5a7f2cc0 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: Notification Message: {'fq_name': ['default-global-system-config', 'cn-jenkins-deploy-platform-juju-os-494-1', 'vhost0'], 'oper': 'UPDATE', 'request-id': 'req-5d0a5fb9-ee6b-4ee5-a938-cf76c763a865', 'type': 'virtual_machine_interface', 'uuid': 'cca6d80d-e05d-4411-8298-d2b2a4db205d'} 03/31/2025 02:19:48.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:49.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:50.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:51.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:51.765 7f0b5932c4e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:19:51.766 7f0b5932c4e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:19:51.766 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:51.766 7f0b5932c4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:19:51.766 7f0b5932c4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:51.766 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:51.766 7f0b5932c4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 elements = { display_name : "cn-jenkins-deploy-platform-juju-os-494-1"fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-juju-os-494-1"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 4242172616174751263, "uuid_lslong": 12476957960846986773}, "enable": true, "created": "2025-03-31T02:15:38.984178", "last_modified": "2025-03-31T02:19:48.344899", "description": null, "user_visible": true, "creator": null}parent_type : "global-system-config"parent_uuid : "0d5441e0-0a93-4574-96f2-d4e0e141b4e1"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}uuid : "3adf3984-b6fc-4e1f-ad27-0f28ddc44e15"virtual_router_dpdk_enabled : falsevirtual_router_ip_address : "10.0.0.17" } deleted = False >> 03/31/2025 02:19:51.766 7f0b5932c4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:51.766 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:51.767 7f0b5932c4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] ContrailConfigTrace: data = << name = default-global-system-config:cn-jenkins-deploy-platform-juju-os-494-1:vhost0 elements = { display_name : "vhost0"fq_name : ["default-global-system-config", "cn-jenkins-deploy-platform-juju-os-494-1", "vhost0"]id_perms : {"permissions": {"owner": "admin", "owner_access": 7, "group": "admin", "group_access": 7, "other_access": 7}, "uuid": {"uuid_mslong": 14746711583981650961, "uuid_lslong": 9410503086102487133}, "enable": true, "created": "2025-03-31T02:15:39.477682", "last_modified": "2025-03-31T02:19:48.675157", "description": null, "user_visible": true, "creator": null}parent_type : "virtual-router"parent_uuid : "3adf3984-b6fc-4e1f-ad27-0f28ddc44e15"perms2 : {"owner": "cloud-admin", "owner_access": 7, "global_access": 0, "share": []}routing_instance_refs : [{"to": ["default-domain", "default-project", "ip-fabric", "ip-fabric"], "attr": {"direction": "both", "vlan_tag": null, "src_mac": null, "dst_mac": null, "mpls_label": null, "service_chain_address": null, "ipv6_service_chain_address": null, "protocol": null}, "uuid": "e10ef3f4-720e-40a6-af67-7f24c5388536"}]uuid : "cca6d80d-e05d-4411-8298-d2b2a4db205d"virtual_machine_interface_disable_policy : truevirtual_machine_interface_mac_addresses : {"mac_address": ["02:cc:a6:d8:0d:e0"]}virtual_network_refs : [{"to": ["default-domain", "default-project", "ip-fabric"], "attr": null, "uuid": "3da78d9c-3d98-4934-a990-647a4cdbfe7f"}] } deleted = False >> 03/31/2025 02:19:51.767 7f0b5932c4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:19:51.767 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:19:51.767 7f0b5932c4e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:19:51.767 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:51.767 7f0b5932c4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:19:51.767 7f0b5932c4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:51.768 7f0b5932c4e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:51.768 7f0b5932c4e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:19:51.768 7f0b5932c4e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:19:52.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:53.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:54.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:55.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:55.765 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:19:56.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:57.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:58.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:19:59.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:00.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:00.767 7f0b5912de10 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:20:00.767 7f0b5912de10 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:20:00.767 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:00.767 7f0b5912de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:20:00.767 7f0b5912de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:00.768 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:20:00.768 7f0b5912de10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:20:00.768 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:00.768 7f0b5912de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:20:00.768 7f0b5912de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:00.768 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:00.769 7f0b5912de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:20:00.769 7f0b5912de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:01.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:02.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:03.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:04.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:04.767 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:20:05.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:06.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:07.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:08.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:09.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:09.770 7f0b5912de10 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:20:09.770 7f0b5912de10 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:20:09.770 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:09.771 7f0b5912de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:20:09.771 7f0b5912de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:09.771 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:20:09.771 7f0b5912de10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:20:09.771 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:09.771 7f0b5912de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:20:09.771 7f0b5912de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:09.771 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:09.772 7f0b5912de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:20:09.772 7f0b5912de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:10.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:11.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:12.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:13.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:13.770 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:20:14.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:16.721 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:17.722 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:18.724 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:18.772 7f0b5912de10 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:20:18.773 7f0b5912de10 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:20:18.773 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:18.773 7f0b5912de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:20:18.773 7f0b5912de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:18.773 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:20:18.774 7f0b5912de10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:20:18.774 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:18.774 7f0b5912de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:20:18.774 7f0b5912de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:18.774 7f0b5912de10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:18.775 7f0b5912de10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:20:18.775 7f0b5912de10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:19.726 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:20.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:21.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:22.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:22.773 7f0b59291e10 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:20:23.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:24.616 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:20:24.616 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:20:24.616 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:20:24.617 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:20:24.617 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:20:24.617 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:20:24.617 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:20:24.617 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:20:24.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:25.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:26.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:27.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:27.774 7f0b59291e10 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:20:27.774 7f0b59291e10 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:20:27.774 7f0b59291e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:27.775 7f0b59291e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:20:27.775 7f0b59291e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:27.775 7f0b59291e10 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:20:27.775 7f0b59291e10 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:20:27.775 7f0b59291e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:27.776 7f0b59291e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:20:27.776 7f0b59291e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:27.776 7f0b59291e10 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:27.776 7f0b59291e10 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:20:27.776 7f0b59291e10 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:28.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:29.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:30.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:31.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:31.774 7f0b59355588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:20:32.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:33.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:34.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:35.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:36.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:36.776 7f0b59355588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:20:36.776 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:20:36.776 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:36.776 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:20:36.776 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:36.777 7f0b59355588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:20:36.777 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:20:36.777 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:36.777 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:20:36.777 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:36.777 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:36.777 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:20:36.777 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:37.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:38.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:39.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:40.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:40.777 7f0b59355588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:20:41.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:42.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:43.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:44.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:45.779 7f0b59355588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:20:45.779 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:20:45.779 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:45.779 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:20:45.779 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:45.779 7f0b59355588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:20:45.780 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:20:45.780 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:45.780 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:20:45.780 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:45.780 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:45.780 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:20:45.780 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:46.721 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:47.723 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:48.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:49.726 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:49.779 7f0b59355588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:20:50.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:51.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:52.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:53.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:54.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:54.781 7f0b59355588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:20:54.781 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:20:54.781 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:54.781 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:20:54.781 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:20:54.781 7f0b59355588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:20:54.782 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:20:54.782 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:54.782 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:20:54.782 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:54.782 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:54.782 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:20:54.782 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:20:55.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:56.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:57.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:58.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:20:58.782 7f0b59355588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:20:59.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:00.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:01.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:02.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:03.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:03.783 7f0b59355588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:21:03.783 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:21:03.784 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:03.784 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:21:03.784 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:03.784 7f0b59355588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:21:03.785 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:21:03.785 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:03.785 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:21:03.785 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:03.785 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:03.785 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:21:03.785 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:04.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:05.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:06.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:07.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:07.784 7f0b59355588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:21:08.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:09.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:10.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:11.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:12.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:12.786 7f0b59355588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:21:12.786 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:21:12.787 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:12.787 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:21:12.787 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:12.787 7f0b59355588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:21:12.787 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:21:12.787 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:12.788 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:21:12.788 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:12.788 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:12.788 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:21:12.788 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:13.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:14.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:16.723 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:16.787 7f0b59355588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:21:17.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:18.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:19.728 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:20.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:21.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:21.788 7f0b59355588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:21:21.789 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:21:21.789 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:21.789 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:21:21.789 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:21.789 7f0b59355588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:21:21.789 7f0b59355588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:21:21.790 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:21.790 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:21:21.790 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:21.790 7f0b59355588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:21.790 7f0b59355588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:21:21.790 7f0b59355588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:22.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:23.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:24.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:25.513 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:21:25.513 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:21:25.513 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:21:25.513 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:21:25.513 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:21:25.513 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:21:25.513 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:21:25.513 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:21:25.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:25.788 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:21:26.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:27.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:28.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:29.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:30.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:30.791 7f0b59157f60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:21:30.791 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:21:30.791 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:30.791 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:21:30.791 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:30.791 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:21:30.792 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:21:30.792 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:30.792 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:21:30.792 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:30.792 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:30.792 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:21:30.792 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:31.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:32.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:33.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:34.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:34.791 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:21:35.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:36.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:37.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:38.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:39.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:39.793 7f0b59157f60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:21:39.793 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:21:39.793 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:39.794 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:21:39.794 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:39.794 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:21:39.794 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:21:39.794 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:39.795 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:21:39.795 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:39.795 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:39.795 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:21:39.795 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:40.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:41.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:42.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:43.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:43.794 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:21:44.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:46.723 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:47.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:48.726 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:48.795 7f0b59157f60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:21:48.795 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:21:48.796 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:48.796 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:21:48.796 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:48.796 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:21:48.796 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:21:48.796 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:48.797 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:21:48.797 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:48.797 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:48.797 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:21:48.797 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:49.727 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:50.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:51.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:52.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:52.795 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:21:53.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:54.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:55.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:56.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:57.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:57.797 7f0b59157f60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:21:57.798 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:21:57.798 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:57.798 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:21:57.798 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:21:57.798 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:21:57.798 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:21:57.799 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:57.799 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:21:57.799 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:57.799 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:57.799 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:21:57.799 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:21:58.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:21:59.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:00.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:01.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:01.798 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:22:02.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:03.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:04.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:05.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:06.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:06.800 7f0b59157f60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:22:06.800 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:22:06.800 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:06.800 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:22:06.800 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:06.800 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:22:06.801 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:22:06.801 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:06.801 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:22:06.801 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:06.801 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:06.801 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:22:06.801 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:07.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:08.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:09.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:10.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:10.800 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:22:11.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:12.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:13.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:14.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:15.802 7f0b59157f60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:22:15.802 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:22:15.802 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:15.802 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:22:15.802 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:15.802 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:22:15.803 7f0b59157f60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:22:15.803 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:15.803 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:22:15.803 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:15.803 7f0b59157f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:15.803 7f0b59157f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:22:15.803 7f0b59157f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:16.725 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:17.726 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:18.728 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:19.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:19.802 7f0b590a2f60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:22:20.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:21.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:22.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:23.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:24.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:24.804 7f0b590a2f60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:22:24.805 7f0b590a2f60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:22:24.805 7f0b590a2f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:24.805 7f0b590a2f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:22:24.805 7f0b590a2f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:24.805 7f0b590a2f60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:22:24.806 7f0b590a2f60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:22:24.806 7f0b590a2f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:24.806 7f0b590a2f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:22:24.806 7f0b590a2f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:24.806 7f0b590a2f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:24.806 7f0b590a2f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:22:24.806 7f0b590a2f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:25.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:26.495 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:22:26.496 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:22:26.496 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:22:26.496 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:22:26.496 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:22:26.496 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:22:26.496 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:22:26.496 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:22:26.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:27.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:28.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:28.805 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:22:29.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:30.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:31.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:32.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:33.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:33.807 7f0b590bc6d8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:22:33.807 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:22:33.807 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:33.807 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:22:33.808 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:33.808 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:22:33.808 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:22:33.808 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:33.808 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:22:33.808 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:33.809 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:33.809 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:22:33.809 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:34.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:35.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:36.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:37.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:37.807 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:22:38.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:39.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:40.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:41.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:42.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:42.809 7f0b590bc6d8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:22:42.809 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:22:42.809 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:42.809 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:22:42.809 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:42.809 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:22:42.810 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:22:42.810 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:42.810 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:22:42.810 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:42.810 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:42.810 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:22:42.810 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:43.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:44.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:46.726 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:46.810 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:22:47.728 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:48.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:49.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:50.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:51.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:51.813 7f0b590bc6d8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:22:51.813 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:22:51.813 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:51.814 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:22:51.814 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:22:51.814 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:22:51.814 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:22:51.814 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:51.815 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:22:51.815 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:51.815 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:51.815 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:22:51.815 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:22:52.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:53.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:54.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:55.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:55.814 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:22:56.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:57.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:58.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:22:59.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:00.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:00.816 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:23:00.816 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:23:00.816 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:00.816 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:23:00.816 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:00.816 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:23:00.817 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:23:00.817 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:00.817 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:23:00.817 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:00.817 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:00.817 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:23:00.817 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:01.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:02.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:03.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:04.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:04.816 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:23:05.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:06.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:07.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:08.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:09.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:09.818 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:23:09.818 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:23:09.818 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:09.818 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:23:09.818 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:09.818 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:23:09.819 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:23:09.819 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:09.819 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:23:09.819 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:09.819 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:09.819 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:23:09.819 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:10.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:11.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:12.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:13.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:13.819 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:23:14.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:16.726 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:17.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:18.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:18.820 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:23:18.821 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:23:18.822 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:18.822 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:23:18.822 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:18.822 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:23:18.822 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:23:18.823 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:18.823 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:23:18.823 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:18.823 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:18.823 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:23:18.823 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:19.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:20.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:21.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:22.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:22.821 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:23:23.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:24.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:25.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:26.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:27.391 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:23:27.391 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:23:27.391 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:23:27.391 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:23:27.391 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:23:27.391 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:23:27.392 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:23:27.392 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:23:27.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:27.823 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:23:27.823 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:23:27.823 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:27.823 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:23:27.823 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:27.823 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:23:27.824 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:23:27.824 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:27.824 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:23:27.824 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:27.824 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:27.824 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:23:27.824 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:28.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:29.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:30.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:31.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:31.823 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:23:32.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:33.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:34.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:35.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:36.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:36.825 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:23:36.826 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:23:36.826 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:36.826 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:23:36.826 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:36.826 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:23:36.827 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:23:36.827 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:36.827 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:23:36.827 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:36.827 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:36.827 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:23:36.827 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:37.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:38.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:39.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:40.731 7f0b5932feb8 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:23:40.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:40.786 7f0b5932feb8 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:23:40.826 7f0b591404e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:23:41.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:42.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:43.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:44.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:45.829 7f0b591404e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:23:45.829 7f0b591404e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:23:45.829 7f0b591404e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:45.829 7f0b591404e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:23:45.829 7f0b591404e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:45.829 7f0b591404e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:23:45.830 7f0b591404e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:23:45.830 7f0b591404e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:45.830 7f0b591404e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:23:45.830 7f0b591404e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:45.830 7f0b591404e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:45.830 7f0b591404e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:23:45.830 7f0b591404e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:46.728 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:47.729 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:48.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:49.324 7f0b5932feb8 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:23:49.380 7f0b5932feb8 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:23:49.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:49.829 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:23:50.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:51.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:52.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:53.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:54.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:54.832 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:23:54.832 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:23:54.832 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:54.832 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:23:54.832 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:23:54.832 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:23:54.833 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:23:54.833 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:54.833 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:23:54.833 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:54.833 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:54.833 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:23:54.834 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:23:55.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:56.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:57.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:58.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:23:58.832 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:23:59.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:00.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:01.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:02.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:03.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:03.834 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:24:03.834 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:24:03.834 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:03.834 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:24:03.834 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:03.834 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:24:03.835 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:24:03.835 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:03.835 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:24:03.835 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:03.835 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:03.835 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:24:03.835 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:04.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:04.843 7f0b5932feb8 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:24:04.893 7f0b5932feb8 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:24:05.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:06.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:07.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:07.834 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:24:08.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:09.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:10.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:11.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:12.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:12.836 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:24:12.837 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:24:12.838 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:12.838 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:24:12.838 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:12.838 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:24:12.839 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:24:12.839 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:12.839 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:24:12.839 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:12.839 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:12.839 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:24:12.839 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:13.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:14.417 7f0b5932feb8 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:24:14.474 7f0b5932feb8 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:24:14.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:16.728 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:16.837 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:24:17.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:18.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:19.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:20.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:21.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:21.840 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:24:21.840 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:24:21.841 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:21.841 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:24:21.841 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:21.841 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:24:21.842 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:24:21.842 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:21.842 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:24:21.842 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:21.842 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:21.842 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:24:21.842 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:22.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:23.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:24.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:25.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:25.841 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:24:26.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:27.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:28.360 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:24:28.360 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:24:28.360 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:24:28.361 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:24:28.361 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:24:28.361 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:24:28.361 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:24:28.361 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:24:28.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:29.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:30.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:30.843 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:24:30.843 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:24:30.843 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:30.843 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:24:30.843 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:30.843 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:24:30.844 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:24:30.844 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:30.844 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:24:30.844 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:30.844 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:30.844 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:24:30.844 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:31.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:32.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:33.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:34.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:34.843 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:24:35.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:36.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:37.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:38.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:39.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:39.845 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:24:39.845 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:24:39.845 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:39.845 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:24:39.845 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:39.845 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:24:39.846 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:24:39.846 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:39.846 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:24:39.846 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:39.846 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:39.846 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:24:39.847 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:40.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:41.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:42.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:43.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:43.846 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:24:44.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:46.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:47.731 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:48.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:48.848 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:24:48.848 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:24:48.848 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:48.848 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:24:48.848 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:48.848 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:24:48.849 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:24:48.849 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:48.849 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:24:48.849 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:48.849 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:48.849 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:24:48.849 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:49.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:50.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:51.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:52.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:52.848 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:24:53.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:54.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:55.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:56.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:57.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:57.850 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:24:57.851 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:24:57.851 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:57.851 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:24:57.851 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:24:57.851 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:24:57.851 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:24:57.852 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:57.852 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:24:57.852 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:57.852 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:57.852 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:24:57.852 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:24:58.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:24:59.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:00.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:01.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:01.850 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:25:02.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:03.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:04.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:05.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:06.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:06.852 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:25:06.852 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:25:06.852 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:06.853 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:25:06.853 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:06.853 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:25:06.853 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:25:06.853 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:06.854 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:25:06.854 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:06.854 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:06.854 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:25:06.854 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:07.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:08.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:09.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:10.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:10.853 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:25:11.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:12.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:13.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:14.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:15.855 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:25:15.855 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:25:15.855 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:15.855 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:25:15.855 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:15.855 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:25:15.856 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:25:15.856 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:15.856 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:25:15.856 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:15.856 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:15.856 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:25:15.856 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:16.730 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:17.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:18.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:19.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:19.855 7f0b59816ac8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:25:20.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:21.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:22.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:23.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:24.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:24.857 7f0b59816ac8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:25:24.858 7f0b59816ac8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:25:24.858 7f0b59816ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:24.858 7f0b59816ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:25:24.858 7f0b59816ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:24.858 7f0b59816ac8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:25:24.858 7f0b59816ac8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:25:24.859 7f0b59816ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:24.859 7f0b59816ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:25:24.859 7f0b59816ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:24.859 7f0b59816ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:24.859 7f0b59816ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:25:24.859 7f0b59816ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:25.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:26.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:27.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:28.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:28.858 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:25:29.248 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:25:29.249 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:25:29.249 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:25:29.249 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:25:29.249 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:25:29.249 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:25:29.249 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:25:29.249 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:25:29.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:30.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:31.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:32.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:33.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:33.860 7f0b591320f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:25:33.860 7f0b591320f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:25:33.860 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:33.860 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:25:33.860 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:33.860 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:25:33.861 7f0b591320f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:25:33.861 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:33.861 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:25:33.861 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:33.861 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:33.861 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:25:33.861 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:34.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:35.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:36.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:37.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:37.860 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:25:38.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:39.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:40.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:41.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:42.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:42.862 7f0b591320f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:25:42.863 7f0b591320f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:25:42.863 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:42.863 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:25:42.863 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:42.863 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:25:42.864 7f0b591320f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:25:42.864 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:42.864 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:25:42.864 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:42.864 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:42.864 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:25:42.864 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:43.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:44.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:46.732 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:46.863 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:25:47.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:48.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:49.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:50.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:51.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:51.865 7f0b591320f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:25:51.865 7f0b591320f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:25:51.866 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:51.866 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:25:51.867 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:25:51.867 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:25:51.867 7f0b591320f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:25:51.867 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:51.867 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:25:51.867 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:51.868 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:51.868 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:25:51.868 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:25:52.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:53.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:54.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:55.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:55.865 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:25:56.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:57.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:58.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:25:59.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:00.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:00.867 7f0b591320f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:26:00.867 7f0b591320f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:26:00.867 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:00.867 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:26:00.867 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:00.867 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:26:00.868 7f0b591320f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:26:00.868 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:00.868 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:26:00.868 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:00.868 7f0b591320f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:00.868 7f0b591320f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:26:00.869 7f0b591320f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:01.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:02.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:03.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:04.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:04.867 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:26:05.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:06.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:07.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:08.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:09.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:09.869 7f0b590bca20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:26:09.869 7f0b590bca20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:26:09.869 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:09.870 7f0b590bca20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:26:09.870 7f0b590bca20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:09.870 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:26:09.870 7f0b590bca20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:26:09.870 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:09.870 7f0b590bca20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:26:09.871 7f0b590bca20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:09.871 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:09.871 7f0b590bca20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:26:09.871 7f0b590bca20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:10.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:11.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:12.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:13.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:13.870 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:26:14.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:16.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:17.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:18.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:18.871 7f0b590bca20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:26:18.871 7f0b590bca20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:26:18.871 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:18.872 7f0b590bca20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:26:18.872 7f0b590bca20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:18.873 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:26:18.873 7f0b590bca20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:26:18.873 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:18.873 7f0b590bca20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:26:18.874 7f0b590bca20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:18.874 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:18.874 7f0b590bca20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:26:18.874 7f0b590bca20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:19.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:20.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:21.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:22.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:22.872 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:26:23.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:24.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:25.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:26.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:27.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:27.874 7f0b590bca20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:26:27.874 7f0b590bca20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:26:27.874 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:27.874 7f0b590bca20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:26:27.874 7f0b590bca20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:27.876 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:26:27.876 7f0b590bca20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:26:27.876 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:27.876 7f0b590bca20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:26:27.877 7f0b590bca20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:27.877 7f0b590bca20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:27.877 7f0b590bca20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:26:27.877 7f0b590bca20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:28.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:29.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:30.140 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:26:30.141 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:26:30.141 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:26:30.141 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:26:30.141 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:26:30.141 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:26:30.141 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:26:30.141 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:26:30.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:31.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:31.875 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:26:32.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:33.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:34.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:35.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:36.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:36.876 7f0b592d2438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:26:36.877 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:26:36.877 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:36.877 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:26:36.877 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:36.877 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:26:36.877 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:26:36.877 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:36.878 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:26:36.878 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:36.878 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:36.878 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:26:36.878 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:37.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:38.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:39.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:40.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:40.877 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:26:41.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:42.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:43.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:44.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:45.879 7f0b592d2438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:26:45.879 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:26:45.879 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:45.879 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:26:45.879 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:45.879 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:26:45.880 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:26:45.880 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:45.880 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:26:45.881 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:45.881 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:45.881 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:26:45.881 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:46.733 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:47.735 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:48.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:49.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:49.879 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:26:50.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:51.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:52.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:53.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:54.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:54.881 7f0b592d2438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:26:54.881 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:26:54.881 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:54.881 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:26:54.881 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:26:54.881 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:26:54.882 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:26:54.882 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:54.882 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:26:54.882 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:54.882 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:54.882 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:26:54.882 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:26:55.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:56.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:57.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:58.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:26:58.881 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:26:59.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:00.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:01.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:02.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:03.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:03.883 7f0b592d2438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:27:03.884 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:27:03.884 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:03.884 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:27:03.884 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:03.884 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:27:03.885 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:27:03.885 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:03.885 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:27:03.885 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:03.885 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:03.886 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:27:03.886 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:04.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:05.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:06.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:07.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:07.884 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:27:08.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:09.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:10.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:11.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:12.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:12.887 7f0b592d2438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:27:12.887 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:27:12.887 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:12.887 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:27:12.887 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:12.887 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:27:12.888 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:27:12.888 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:12.888 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:27:12.888 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:12.888 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:12.888 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:27:12.888 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:13.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:14.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:16.734 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:16.887 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:27:17.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:18.737 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:19.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:20.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:21.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:21.889 7f0b592d2438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:27:21.890 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:27:21.890 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:21.890 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:27:21.891 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:21.891 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:27:21.892 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:27:21.892 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:21.892 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:27:21.892 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:21.892 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:21.892 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:27:21.892 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:22.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:23.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:24.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:25.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:25.890 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:27:26.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:27.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:28.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:29.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:30.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:30.892 7f0b592d2438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:27:30.892 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:27:30.892 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:30.893 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:27:30.893 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:30.893 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:27:30.893 7f0b592d2438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:27:30.893 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:30.894 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:27:30.894 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:30.894 7f0b592d2438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:30.894 7f0b592d2438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:27:30.894 7f0b592d2438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:31.028 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:27:31.028 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:27:31.028 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:27:31.028 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:27:31.029 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:27:31.029 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:27:31.029 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:27:31.029 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:27:31.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:32.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:33.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:34.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:34.892 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:27:35.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:36.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:37.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:38.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:39.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:39.894 7f0b595a4780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:27:39.895 7f0b595a4780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:27:39.895 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:39.895 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:27:39.895 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:39.895 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:27:39.896 7f0b595a4780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:27:39.896 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:39.896 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:27:39.896 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:39.896 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:39.896 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:27:39.897 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:40.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:41.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:42.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:43.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:43.895 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:27:44.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:46.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:47.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:48.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:48.896 7f0b595a4780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:27:48.896 7f0b595a4780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:27:48.896 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:48.897 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:27:48.897 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:48.897 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:27:48.897 7f0b595a4780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:27:48.897 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:48.897 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:27:48.898 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:48.898 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:48.898 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:27:48.898 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:49.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:50.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:51.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:52.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:52.897 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:27:53.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:54.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:55.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:56.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:57.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:57.899 7f0b595a4780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:27:57.899 7f0b595a4780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:27:57.899 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:57.899 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:27:57.900 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:27:57.900 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:27:57.900 7f0b595a4780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:27:57.900 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:57.900 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:27:57.900 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:57.900 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:57.900 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:27:57.900 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:27:58.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:27:59.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:00.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:01.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:01.899 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:28:02.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:03.120 7f0b592c1390 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:28:03.172 7f0b592c1390 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:28:03.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:04.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:05.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:06.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:06.902 7f0b595a4780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:28:06.902 7f0b595a4780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:28:06.902 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:06.902 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:28:06.902 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:06.903 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:28:06.903 7f0b595a4780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:28:06.903 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:06.903 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:28:06.903 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:06.903 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:06.904 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:28:06.904 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:07.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:08.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:09.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:10.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:10.902 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:28:11.625 7f0b590c1588 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:28:11.676 7f0b590c1588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:28:11.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:12.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:13.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:14.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:15.904 7f0b595a4780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:28:15.904 7f0b595a4780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:28:15.904 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:15.904 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:28:15.904 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:15.904 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:28:15.905 7f0b595a4780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:28:15.905 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:15.905 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:28:15.905 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:15.905 7f0b595a4780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:15.905 7f0b595a4780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:28:15.905 7f0b595a4780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:16.736 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:17.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:18.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:19.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:19.904 7f0b5934d828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:28:20.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:21.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:22.725 7f0b5936b978 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:28:22.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:22.776 7f0b595d1cc0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:28:23.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:24.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:24.905 7f0b5934d828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:28:24.906 7f0b5934d828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:28:24.906 7f0b5934d828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:24.906 7f0b5934d828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:28:24.906 7f0b5934d828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:24.906 7f0b5934d828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:28:24.906 7f0b5934d828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:28:24.907 7f0b5934d828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:24.907 7f0b5934d828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:28:24.907 7f0b5934d828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:24.907 7f0b5934d828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:24.907 7f0b5934d828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:28:24.907 7f0b5934d828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:25.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:26.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:27.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:28.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:28.906 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:28:29.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:30.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:31.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:31.946 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:28:31.946 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:28:31.946 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:28:31.946 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:28:31.946 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:28:31.946 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:28:31.946 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:28:31.947 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:28:32.293 7f0b5914af60 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:28:32.344 7f0b5914af60 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:28:32.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:33.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:33.908 7f0b5932feb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:28:33.909 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:28:33.909 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:33.909 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:28:33.909 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:33.909 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:28:33.909 7f0b5932feb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:28:33.910 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:33.910 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:28:33.910 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:33.910 7f0b5932feb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:33.910 7f0b5932feb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:28:33.910 7f0b5932feb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:34.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:35.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:36.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:37.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:37.909 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:28:38.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:39.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:40.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:41.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:42.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:42.911 7f0b592c1198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:28:42.912 7f0b592c1198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:28:42.912 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:42.912 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:28:42.912 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:42.912 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:28:42.913 7f0b592c1198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:28:42.913 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:42.913 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:28:42.913 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:42.913 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:42.913 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:28:42.913 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:43.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:44.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:46.738 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:46.912 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:28:47.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:48.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:49.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:50.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:51.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:51.913 7f0b592c1198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:28:51.914 7f0b592c1198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:28:51.914 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:51.914 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:28:51.914 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:28:51.914 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:28:51.915 7f0b592c1198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:28:51.915 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:51.915 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:28:51.915 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:51.915 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:51.916 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:28:51.916 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:28:52.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:53.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:54.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:55.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:55.914 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:28:56.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:57.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:58.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:28:59.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:00.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:00.916 7f0b592c1198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:29:00.916 7f0b592c1198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:29:00.917 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:00.917 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:29:00.917 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:00.917 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:29:00.917 7f0b592c1198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:29:00.918 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:00.918 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:29:00.918 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:00.918 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:00.918 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:29:00.918 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:01.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:02.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:03.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:04.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:04.917 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:29:05.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:06.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:07.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:08.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:09.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:09.919 7f0b592c1198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:29:09.919 7f0b592c1198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:29:09.919 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:09.919 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:29:09.919 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:09.919 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:29:09.920 7f0b592c1198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:29:09.920 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:09.920 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:29:09.920 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:09.920 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:09.920 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:29:09.920 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:10.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:11.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:12.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:13.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:13.919 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:29:14.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:16.739 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:17.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:18.741 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:18.920 7f0b592c1198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:29:18.921 7f0b592c1198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:29:18.921 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:18.921 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:29:18.921 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:18.921 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:29:18.922 7f0b592c1198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:29:18.922 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:18.922 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:29:18.922 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:18.923 7f0b592c1198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:18.923 7f0b592c1198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:29:18.923 7f0b592c1198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:19.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:20.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:21.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:22.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:22.921 7f0b5912bf60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:29:23.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:24.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:25.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:26.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:27.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:27.923 7f0b5912bf60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:29:27.923 7f0b5912bf60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:29:27.924 7f0b5912bf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:27.924 7f0b5912bf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:29:27.924 7f0b5912bf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:27.924 7f0b5912bf60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:29:27.924 7f0b5912bf60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:29:27.924 7f0b5912bf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:27.924 7f0b5912bf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:29:27.925 7f0b5912bf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:27.925 7f0b5912bf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:27.925 7f0b5912bf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:29:27.925 7f0b5912bf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:28.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:29.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:30.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:31.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:31.923 7f0b5912bf60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:29:32.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:32.825 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:29:32.825 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:29:32.825 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:29:32.825 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:29:32.825 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:29:32.825 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:29:32.825 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:29:32.826 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:29:33.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:34.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:35.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:36.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:36.925 7f0b5912bf60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:29:36.925 7f0b5912bf60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:29:36.926 7f0b5912bf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:36.926 7f0b5912bf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:29:36.926 7f0b5912bf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:36.926 7f0b5912bf60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:29:36.926 7f0b5912bf60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:29:36.926 7f0b5912bf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:36.927 7f0b5912bf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:29:36.927 7f0b5912bf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:36.927 7f0b5912bf60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:36.927 7f0b5912bf60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:29:36.927 7f0b5912bf60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:37.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:38.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:39.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:40.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:40.926 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:29:41.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:42.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:43.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:44.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:45.928 7f0b591574e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:29:45.928 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:29:45.928 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:45.928 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:29:45.928 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:45.928 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:29:45.929 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:29:45.929 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:45.929 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:29:45.929 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:45.929 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:45.930 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:29:45.931 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:46.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:47.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:48.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:49.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:49.929 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:29:50.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:51.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:52.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:53.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:54.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:54.933 7f0b591574e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:29:54.933 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:29:54.933 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:54.933 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:29:54.933 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:29:54.933 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:29:54.934 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:29:54.934 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:54.934 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:29:54.934 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:54.934 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:54.934 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:29:54.934 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:29:55.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:56.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:57.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:58.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:29:58.933 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:29:59.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:00.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:01.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:02.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:03.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:03.936 7f0b591574e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:30:03.936 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:30:03.936 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:03.936 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:30:03.936 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:03.936 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:30:03.937 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:30:03.937 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:03.937 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:30:03.937 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:03.938 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:03.938 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:30:03.938 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:04.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:05.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:06.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:07.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:07.936 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:30:08.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:09.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:10.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:11.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:12.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:12.938 7f0b591574e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:30:12.938 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:30:12.938 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:12.938 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:30:12.939 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:12.939 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:30:12.939 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:30:12.939 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:12.939 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:30:12.939 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:12.939 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:12.940 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:30:12.940 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:13.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:14.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:16.740 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:16.938 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:30:17.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:18.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:19.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:20.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:21.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:21.941 7f0b591574e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:30:21.943 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:30:21.943 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:21.944 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:30:21.945 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:21.945 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:30:21.945 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:30:21.945 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:21.946 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:30:21.946 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:21.946 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:21.946 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:30:21.946 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:22.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:23.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:24.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:25.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:25.941 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:30:26.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:27.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:28.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:29.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:30.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:30.943 7f0b591574e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:30:30.943 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:30:30.943 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:30.944 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:30:30.944 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:30.944 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:30:30.944 7f0b591574e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:30:30.944 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:30.945 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:30:30.945 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:30.945 7f0b591574e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:30.945 7f0b591574e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:30:30.945 7f0b591574e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:31.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:32.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:33.745 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:30:33.745 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:30:33.745 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:30:33.745 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:30:33.745 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:30:33.746 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:30:33.746 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:30:33.746 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:30:33.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:34.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:34.944 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:30:35.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:36.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:37.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:38.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:39.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:39.946 7f0b5914ab70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:30:39.947 7f0b5914ab70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:30:39.947 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:39.947 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:30:39.947 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:39.947 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:30:39.948 7f0b5914ab70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:30:39.948 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:39.948 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:30:39.948 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:39.948 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:39.948 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:30:39.948 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:40.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:41.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:42.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:43.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:43.947 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:30:44.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:46.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:47.744 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:48.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:48.949 7f0b5914ab70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:30:48.949 7f0b5914ab70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:30:48.949 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:48.949 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:30:48.950 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:48.950 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:30:48.950 7f0b5914ab70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:30:48.950 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:48.950 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:30:48.951 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:48.951 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:48.951 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:30:48.951 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:49.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:50.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:51.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:52.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:52.949 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:30:53.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:54.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:55.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:56.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:57.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:57.951 7f0b5914ab70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:30:57.952 7f0b5914ab70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:30:57.952 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:57.952 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:30:57.952 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:30:57.952 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:30:57.953 7f0b5914ab70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:30:57.953 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:57.953 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:30:57.953 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:57.953 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:57.953 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:30:57.953 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:30:58.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:30:59.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:00.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:01.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:01.952 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:31:02.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:03.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:04.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:05.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:06.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:06.955 7f0b5914ab70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:31:06.955 7f0b5914ab70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:31:06.955 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:06.955 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:31:06.956 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:06.956 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:31:06.956 7f0b5914ab70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:31:06.956 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:06.956 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:31:06.956 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:06.957 7f0b5914ab70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:06.957 7f0b5914ab70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:31:06.957 7f0b5914ab70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:07.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:08.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:09.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:10.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:10.955 7f0b59292eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:31:11.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:12.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:13.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:14.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:15.958 7f0b59292eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:31:15.958 7f0b59292eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:31:15.958 7f0b59292eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:15.958 7f0b59292eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:31:15.958 7f0b59292eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:15.958 7f0b59292eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:31:15.959 7f0b59292eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:31:15.959 7f0b59292eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:15.959 7f0b59292eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:31:15.959 7f0b59292eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:15.959 7f0b59292eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:15.959 7f0b59292eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:31:15.959 7f0b59292eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:16.742 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:17.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:18.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:19.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:19.958 7f0b5a1c8198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:31:20.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:21.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:22.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:23.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:24.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:24.960 7f0b5a1c8198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:31:24.960 7f0b5a1c8198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:31:24.960 7f0b5a1c8198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:24.960 7f0b5a1c8198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:31:24.961 7f0b5a1c8198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:24.961 7f0b5a1c8198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:31:24.961 7f0b5a1c8198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:31:24.961 7f0b5a1c8198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:24.961 7f0b5a1c8198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:31:24.961 7f0b5a1c8198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:24.961 7f0b5a1c8198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:24.962 7f0b5a1c8198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:31:24.962 7f0b5a1c8198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:25.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:26.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:27.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:28.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:28.961 7f0b5a1c8198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:31:29.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:30.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:31.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:32.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:33.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:33.963 7f0b5a1c8198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:31:33.963 7f0b5a1c8198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:31:33.963 7f0b5a1c8198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:33.963 7f0b5a1c8198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:31:33.963 7f0b5a1c8198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:33.964 7f0b5a1c8198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:31:33.964 7f0b5a1c8198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:31:33.964 7f0b5a1c8198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:33.964 7f0b5a1c8198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:31:33.964 7f0b5a1c8198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:33.964 7f0b5a1c8198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:33.965 7f0b5a1c8198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:31:33.965 7f0b5a1c8198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:34.627 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:31:34.628 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:31:34.628 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:31:34.628 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:31:34.628 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:31:34.628 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:31:34.628 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:31:34.628 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:31:34.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:35.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:36.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:37.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:37.964 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:31:38.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:39.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:40.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:41.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:42.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:42.966 7f0b592618d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:31:42.966 7f0b592618d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:31:42.966 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:42.966 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:31:42.967 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:42.967 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:31:42.967 7f0b592618d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:31:42.967 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:42.967 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:31:42.967 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:42.968 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:42.968 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:31:42.968 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:43.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:44.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:46.743 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:46.967 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:31:47.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:48.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:49.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:50.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:51.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:51.969 7f0b592618d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:31:51.969 7f0b592618d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:31:51.969 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:51.969 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:31:51.969 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:31:51.969 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:31:51.970 7f0b592618d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:31:51.970 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:51.970 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:31:51.970 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:51.970 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:51.970 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:31:51.970 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:31:52.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:53.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:54.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:55.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:55.969 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:31:56.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:57.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:58.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:31:59.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:00.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:00.971 7f0b592618d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:32:00.971 7f0b592618d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:32:00.971 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:00.971 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:32:00.971 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:00.971 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:32:00.972 7f0b592618d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:32:00.972 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:00.972 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:32:00.972 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:00.972 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:00.972 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:32:00.972 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:01.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:02.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:03.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:04.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:04.971 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:32:05.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:06.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:07.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:08.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:09.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:09.973 7f0b592618d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:32:09.973 7f0b592618d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:32:09.974 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:09.974 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:32:09.974 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:09.974 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:32:09.974 7f0b592618d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:32:09.974 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:09.975 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:32:09.975 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:09.975 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:09.975 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:32:09.975 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:10.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:11.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:12.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:13.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:13.974 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:32:14.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:16.745 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:17.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:18.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:18.976 7f0b592618d0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:32:18.976 7f0b592618d0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:32:18.977 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:18.977 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:32:18.977 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:18.977 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:32:18.977 7f0b592618d0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:32:18.977 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:18.978 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:32:18.978 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:18.978 7f0b592618d0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:18.978 7f0b592618d0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:32:18.978 7f0b592618d0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:19.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:20.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:21.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:22.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:22.977 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:32:23.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:24.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:25.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:26.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:27.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:27.979 7f0b595a1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:32:27.979 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:32:27.979 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:27.980 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:32:27.980 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:27.980 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:32:27.980 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:32:27.980 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:27.980 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:32:27.980 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:27.981 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:27.981 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:32:27.981 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:28.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:29.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:30.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:31.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:31.980 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:32:32.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:33.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:34.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:35.532 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:32:35.532 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:32:35.532 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:32:35.533 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:32:35.533 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:32:35.533 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:32:35.533 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:32:35.533 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:32:35.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:36.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:36.983 7f0b595a1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:32:36.983 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:32:36.983 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:36.983 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:32:36.983 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:36.983 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:32:36.984 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:32:36.984 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:36.984 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:32:36.984 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:36.984 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:36.984 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:32:36.984 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:37.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:38.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:39.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:40.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:40.983 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:32:41.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:42.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:43.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:44.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:45.985 7f0b595a1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:32:45.985 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:32:45.985 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:45.985 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:32:45.985 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:45.986 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:32:45.986 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:32:45.986 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:45.986 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:32:45.986 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:45.986 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:45.987 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:32:45.987 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:46.746 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:47.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:48.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:49.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:49.986 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:32:50.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:51.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:52.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:53.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:54.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:54.988 7f0b595a1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:32:54.988 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:32:54.988 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:54.988 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:32:54.988 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:32:54.988 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:32:54.989 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:32:54.989 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:54.989 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:32:54.989 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:54.989 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:54.989 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:32:54.989 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:32:55.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:56.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:57.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:58.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:32:58.988 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:32:59.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:00.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:01.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:02.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:03.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:03.990 7f0b595a1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:33:03.990 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:33:03.990 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:03.990 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:33:03.990 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:03.990 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:33:03.991 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:33:03.991 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:03.991 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:33:03.991 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:03.991 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:03.991 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:33:03.991 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:04.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:05.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:06.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:07.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:07.990 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:33:08.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:09.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:10.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:11.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:12.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:12.992 7f0b595a1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:33:12.992 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:33:12.992 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:12.993 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:33:12.993 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:12.993 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:33:12.994 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:33:12.994 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:12.994 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:33:12.994 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:12.995 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:12.995 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:33:12.996 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:13.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:14.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:16.747 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:16.993 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:33:17.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:18.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:19.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:20.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:21.489 7f0b59067c18 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:33:21.540 7f0b5ec8fe10 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:33:21.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:21.995 7f0b595a1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:33:21.995 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:33:21.996 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:21.996 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:33:21.996 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:21.996 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:33:21.996 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:33:21.997 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:21.997 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:33:21.997 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:21.997 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:21.997 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:33:21.997 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:22.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:23.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:24.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:25.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:25.996 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:33:26.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:27.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:28.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:29.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:30.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:30.998 7f0b595a1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:33:30.998 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:33:30.998 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:30.998 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:33:30.999 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:30.999 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:33:30.999 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:33:30.999 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:30.999 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:33:30.999 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:31.000 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:31.000 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:33:31.000 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:31.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:32.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:33.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:34.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:34.998 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:33:35.517 7f0b59326b70 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:33:35.570 7f0b595774e0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:33:35.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:36.420 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:33:36.420 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:33:36.420 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:33:36.420 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:33:36.420 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:33:36.420 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:33:36.420 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:33:36.420 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:33:36.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:37.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:38.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:39.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:40.000 7f0b595a1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:33:40.000 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:33:40.000 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:40.000 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:33:40.001 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:40.001 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:33:40.001 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:33:40.001 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:40.001 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:33:40.001 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:40.001 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:40.002 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:33:40.002 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:40.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:41.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:42.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:43.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:44.000 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:33:44.101 7f0b5a808438 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:33:44.152 7f0b5a808438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:33:44.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:46.748 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:47.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:48.750 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:49.002 7f0b595a1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:33:49.002 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:33:49.003 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:49.003 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:33:49.003 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:49.003 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:33:49.003 7f0b595a1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:33:49.003 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:49.004 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:33:49.004 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:49.004 7f0b595a1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:49.004 7f0b595a1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:33:49.004 7f0b595a1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:49.752 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:50.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:51.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:52.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:53.003 7f0b590cdb70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:33:53.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:54.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:55.125 7f0b5a808438 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:33:55.176 7f0b5a808438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:33:55.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:56.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:57.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:58.005 7f0b590cdb70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:33:58.005 7f0b590cdb70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:33:58.005 7f0b590cdb70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:58.006 7f0b590cdb70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:33:58.006 7f0b590cdb70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:33:58.006 7f0b590cdb70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:33:58.006 7f0b590cdb70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:33:58.006 7f0b590cdb70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:58.006 7f0b590cdb70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:33:58.007 7f0b590cdb70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:58.007 7f0b590cdb70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:58.007 7f0b590cdb70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:33:58.007 7f0b590cdb70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:33:58.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:33:59.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:00.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:01.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:02.006 7f0b597fbd68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:34:02.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:03.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:04.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:05.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:06.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:07.008 7f0b597fbd68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:34:07.009 7f0b597fbd68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:34:07.009 7f0b597fbd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:07.009 7f0b597fbd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:34:07.009 7f0b597fbd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:07.009 7f0b597fbd68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:34:07.009 7f0b597fbd68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:34:07.010 7f0b597fbd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:07.010 7f0b597fbd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:34:07.010 7f0b597fbd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:07.010 7f0b597fbd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:07.010 7f0b597fbd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:34:07.010 7f0b597fbd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:07.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:08.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:09.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:10.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:11.009 7f0b597fbd68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:34:11.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:12.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:13.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:14.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:16.011 7f0b597fbd68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:34:16.011 7f0b597fbd68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:34:16.011 7f0b597fbd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:16.012 7f0b597fbd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:34:16.012 7f0b597fbd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:16.012 7f0b597fbd68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:34:16.012 7f0b597fbd68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:34:16.012 7f0b597fbd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:16.013 7f0b597fbd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:34:16.013 7f0b597fbd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:16.013 7f0b597fbd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:16.013 7f0b597fbd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:34:16.013 7f0b597fbd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:16.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:17.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:18.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:19.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:20.011 7f0b592c0390 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:34:20.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:21.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:22.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:23.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:24.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:25.017 7f0b592c0390 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:34:25.017 7f0b592c0390 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:34:25.017 7f0b592c0390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:25.017 7f0b592c0390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:34:25.017 7f0b592c0390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:25.017 7f0b592c0390 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:34:25.018 7f0b592c0390 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:34:25.018 7f0b592c0390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:25.018 7f0b592c0390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:34:25.018 7f0b592c0390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:25.018 7f0b592c0390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:25.018 7f0b592c0390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:34:25.018 7f0b592c0390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:25.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:26.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:27.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:28.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:29.019 7f0b592c0390 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:34:29.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:30.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:31.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:32.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:33.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:34.028 7f0b592c0390 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:34:34.028 7f0b592c0390 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:34:34.028 7f0b592c0390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:34.029 7f0b592c0390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:34:34.029 7f0b592c0390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:34.029 7f0b592c0390 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:34:34.029 7f0b592c0390 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:34:34.029 7f0b592c0390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:34.029 7f0b592c0390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:34:34.030 7f0b592c0390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:34.030 7f0b592c0390 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:34.030 7f0b592c0390 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:34:34.030 7f0b592c0390 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:34.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:35.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:36.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:37.400 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:34:37.400 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:34:37.400 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:34:37.400 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:34:37.401 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:34:37.401 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:34:37.401 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:34:37.401 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:34:37.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:38.029 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:34:38.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:39.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:40.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:41.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:42.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:43.031 7f0b592c1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:34:43.031 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:34:43.031 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:43.032 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:34:43.032 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:43.032 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:34:43.032 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:34:43.032 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:43.032 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:34:43.032 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:43.033 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:43.033 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:34:43.033 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:43.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:44.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:46.749 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:47.032 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:34:47.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:48.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:49.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:50.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:51.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:52.035 7f0b592c1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:34:52.035 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:34:52.035 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:52.035 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:34:52.035 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:34:52.035 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:34:52.036 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:34:52.036 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:52.036 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:34:52.036 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:52.036 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:52.036 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:34:52.036 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:34:52.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:53.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:54.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:55.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:56.036 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:34:56.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:57.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:58.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:34:59.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:00.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:01.038 7f0b592c1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:35:01.038 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:35:01.038 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:01.039 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:35:01.039 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:01.039 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:35:01.039 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:35:01.039 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:01.039 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:35:01.039 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:01.040 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:01.040 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:35:01.040 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:01.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:02.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:03.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:04.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:05.038 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:35:05.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:06.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:07.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:08.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:09.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:10.041 7f0b592c1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:35:10.041 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:35:10.041 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:10.042 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:35:10.042 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:10.042 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:35:10.042 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:35:10.042 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:10.042 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:35:10.042 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:10.043 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:10.043 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:35:10.043 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:10.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:11.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:12.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:13.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:14.041 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:35:14.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:16.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:17.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:18.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:19.043 7f0b592c1d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:35:19.043 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:35:19.043 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:19.044 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:35:19.044 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:19.044 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:35:19.044 7f0b592c1d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:35:19.044 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:19.045 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:35:19.046 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:19.046 7f0b592c1d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:19.046 7f0b592c1d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:35:19.046 7f0b592c1d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:19.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:20.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:21.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:22.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:23.044 7f0b59350780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:35:23.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:24.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:25.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:26.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:27.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:28.046 7f0b59350780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:35:28.046 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:35:28.047 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:28.047 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:35:28.047 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:28.047 7f0b59350780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:35:28.048 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:35:28.048 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:28.048 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:35:28.048 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:28.048 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:28.048 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:35:28.048 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:28.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:29.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:30.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:31.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:32.047 7f0b59350780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:35:32.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:33.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:34.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:35.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:36.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:37.049 7f0b59350780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:35:37.049 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:35:37.049 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:37.049 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:35:37.049 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:37.049 7f0b59350780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:35:37.050 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:35:37.050 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:37.050 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:35:37.050 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:37.050 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:37.050 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:35:37.050 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:37.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:38.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:35:38.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:35:38.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:35:38.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:35:38.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:35:38.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:35:38.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:35:38.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:35:38.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:39.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:40.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:41.049 7f0b59131eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:35:41.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:42.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:43.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:44.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:46.051 7f0b59131eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:35:46.051 7f0b59131eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:35:46.051 7f0b59131eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:46.051 7f0b59131eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:35:46.052 7f0b59131eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:46.052 7f0b59131eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:35:46.052 7f0b59131eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:35:46.052 7f0b59131eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:46.052 7f0b59131eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:35:46.052 7f0b59131eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:46.052 7f0b59131eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:46.052 7f0b59131eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:35:46.053 7f0b59131eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:46.751 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:47.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:48.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:49.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:50.052 7f0b59131eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:35:50.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:51.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:52.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:53.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:54.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:55.054 7f0b59131eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:35:55.055 7f0b59131eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:35:55.055 7f0b59131eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:55.055 7f0b59131eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:35:55.055 7f0b59131eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:35:55.055 7f0b59131eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:35:55.056 7f0b59131eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:35:55.056 7f0b59131eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:55.056 7f0b59131eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:35:55.056 7f0b59131eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:55.056 7f0b59131eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:55.056 7f0b59131eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:35:55.056 7f0b59131eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:35:55.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:56.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:57.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:58.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:35:59.055 7f0b59350780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:35:59.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:00.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:01.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:02.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:03.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:04.057 7f0b59350780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:36:04.058 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:36:04.058 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:04.058 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:36:04.058 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:04.058 7f0b59350780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:36:04.058 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:36:04.059 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:04.059 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:36:04.059 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:04.059 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:04.059 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:36:04.059 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:04.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:05.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:06.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:07.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:08.058 7f0b59350780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:36:08.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:09.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:10.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:11.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:12.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:13.060 7f0b59350780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:36:13.060 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:36:13.061 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:13.061 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:36:13.061 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:13.061 7f0b59350780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:36:13.061 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:36:13.062 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:13.062 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:36:13.062 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:13.062 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:13.062 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:36:13.062 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:13.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:14.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:16.753 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:17.061 7f0b59350780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:36:17.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:18.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:19.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:20.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:21.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:22.063 7f0b59350780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:36:22.063 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:36:22.063 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:22.064 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:36:22.064 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:22.064 7f0b59350780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:36:22.064 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:36:22.064 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:22.064 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:36:22.065 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:22.065 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:22.065 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:36:22.065 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:22.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:23.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:24.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:25.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:26.064 7f0b59350780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:36:26.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:27.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:28.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:29.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:30.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:31.066 7f0b59350780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:36:31.066 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:36:31.066 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:31.066 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:36:31.066 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:31.066 7f0b59350780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:36:31.067 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:36:31.067 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:31.067 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:36:31.067 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:31.067 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:31.067 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:36:31.067 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:31.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:32.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:33.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:34.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:35.066 7f0b59350780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:36:35.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:36.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:37.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:38.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:39.187 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:36:39.188 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:36:39.188 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:36:39.188 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:36:39.188 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:36:39.188 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:36:39.188 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:36:39.188 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:36:39.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:40.068 7f0b59350780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:36:40.069 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:36:40.069 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:40.069 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:36:40.069 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:40.069 7f0b59350780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:36:40.070 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:36:40.070 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:40.070 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:36:40.070 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:40.070 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:40.070 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:36:40.070 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:40.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:41.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:42.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:43.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:44.070 7f0b59350780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:36:44.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:46.754 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:47.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:48.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:49.072 7f0b59350780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:36:49.072 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:36:49.072 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:49.072 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:36:49.072 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:49.072 7f0b59350780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:36:49.073 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:36:49.073 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:49.073 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:36:49.073 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:49.073 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:49.073 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:36:49.073 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:49.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:50.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:51.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:52.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:53.073 7f0b59350780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:36:53.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:54.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:55.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:56.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:57.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:58.074 7f0b59350780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:36:58.075 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:36:58.075 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:58.075 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:36:58.075 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:36:58.075 7f0b59350780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:36:58.075 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:36:58.076 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:58.076 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:36:58.076 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:58.076 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:58.076 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:36:58.076 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:36:58.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:36:59.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:00.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:01.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:02.075 7f0b59350780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:37:02.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:03.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:04.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:05.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:06.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:07.077 7f0b59350780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:37:07.077 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:37:07.077 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:07.077 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:37:07.077 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:07.077 7f0b59350780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:37:07.078 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:37:07.078 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:07.078 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:37:07.078 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:07.078 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:07.078 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:37:07.078 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:07.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:08.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:09.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:10.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:11.078 7f0b59350780 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:37:11.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:12.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:13.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:14.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:16.080 7f0b59350780 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:37:16.080 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:37:16.080 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:16.080 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:37:16.080 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:16.081 7f0b59350780 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:37:16.081 7f0b59350780 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:37:16.081 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:16.081 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:37:16.083 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:16.083 7f0b59350780 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:16.083 7f0b59350780 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:37:16.083 7f0b59350780 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:16.755 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:17.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:18.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:19.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:20.080 7f0b59068eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:37:20.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:21.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:22.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:23.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:24.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:25.082 7f0b59068eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:37:25.083 7f0b59068eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:37:25.083 7f0b59068eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:25.083 7f0b59068eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:37:25.083 7f0b59068eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:25.083 7f0b59068eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:37:25.084 7f0b59068eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:37:25.084 7f0b59068eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:25.084 7f0b59068eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:37:25.084 7f0b59068eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:25.084 7f0b59068eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:25.084 7f0b59068eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:37:25.084 7f0b59068eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:25.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:26.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:27.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:28.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:29.083 7f0b59068eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:37:29.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:30.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:31.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:32.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:33.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:34.085 7f0b59068eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:37:34.085 7f0b59068eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:37:34.086 7f0b59068eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:34.086 7f0b59068eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:37:34.086 7f0b59068eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:34.086 7f0b59068eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:37:34.086 7f0b59068eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:37:34.087 7f0b59068eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:34.087 7f0b59068eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:37:34.087 7f0b59068eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:34.087 7f0b59068eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:34.087 7f0b59068eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:37:34.087 7f0b59068eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:34.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:35.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:36.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:37.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:38.086 7f0b590af438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:37:38.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:39.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:40.153 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:37:40.153 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:37:40.153 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:37:40.153 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:37:40.154 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:37:40.154 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:37:40.154 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:37:40.154 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:37:40.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:41.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:42.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:43.088 7f0b590af438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:37:43.088 7f0b590af438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:37:43.088 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:43.088 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:37:43.088 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:43.088 7f0b590af438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:37:43.089 7f0b590af438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:37:43.089 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:43.089 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:37:43.089 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:43.089 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:43.089 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:37:43.089 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:43.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:44.175 7f0b590af438 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:37:44.228 7f0b590af438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:37:44.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:46.756 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:47.089 7f0b590af438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:37:47.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:48.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:49.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:50.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:51.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:52.090 7f0b590af438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:37:52.091 7f0b590af438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:37:52.091 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:52.091 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:37:52.091 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:37:52.092 7f0b590af438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:37:52.092 7f0b590af438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:37:52.092 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:52.092 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:37:52.093 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:52.093 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:52.093 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:37:52.093 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:37:52.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:53.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:54.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:55.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:56.092 7f0b590af438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:37:56.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:57.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:58.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:37:59.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:00.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:01.094 7f0b590af438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:01.094 7f0b590af438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:01.094 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:01.094 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:01.094 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:01.094 7f0b590af438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:01.095 7f0b590af438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:01.095 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:01.095 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:01.095 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:01.095 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:01.095 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:38:01.095 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:01.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:02.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:03.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:04.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:05.094 7f0b590af438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:05.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:06.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:07.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:08.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:09.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:10.097 7f0b590af438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:10.097 7f0b590af438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:10.097 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:10.097 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:10.098 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:10.098 7f0b590af438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:10.098 7f0b590af438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:10.098 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:10.099 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:10.099 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:10.099 7f0b590af438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:10.099 7f0b590af438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:38:10.099 7f0b590af438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:10.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:11.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:12.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:13.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:14.098 7f0b595c7cc0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:14.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:16.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:17.758 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:18.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:19.100 7f0b595c7cc0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:19.100 7f0b595c7cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:19.100 7f0b595c7cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:19.100 7f0b595c7cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:19.100 7f0b595c7cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:19.101 7f0b595c7cc0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:19.101 7f0b595c7cc0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:19.101 7f0b595c7cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:19.101 7f0b595c7cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:19.101 7f0b595c7cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:19.101 7f0b595c7cc0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:19.101 7f0b595c7cc0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:38:19.102 7f0b595c7cc0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:19.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:20.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:21.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:22.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:23.101 7f0b5932a0f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:23.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:24.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:25.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:26.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:27.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:28.103 7f0b5932a0f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:28.103 7f0b5932a0f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:28.103 7f0b5932a0f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:28.103 7f0b5932a0f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:28.104 7f0b5932a0f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:28.104 7f0b5932a0f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:28.104 7f0b5932a0f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:28.104 7f0b5932a0f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:28.104 7f0b5932a0f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:28.104 7f0b5932a0f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:28.104 7f0b5932a0f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:28.104 7f0b5932a0f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:38:28.104 7f0b5932a0f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:28.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:29.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:30.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:31.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:32.104 7f0b5932a0f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:32.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:33.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:34.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:35.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:36.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:37.106 7f0b5932a0f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:37.107 7f0b5932a0f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:37.107 7f0b5932a0f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:37.107 7f0b5932a0f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:37.107 7f0b5932a0f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:37.107 7f0b5932a0f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:37.107 7f0b5932a0f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:37.107 7f0b5932a0f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:37.108 7f0b5932a0f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:37.108 7f0b5932a0f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:37.108 7f0b5932a0f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:37.108 7f0b5932a0f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:38:37.108 7f0b5932a0f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:37.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:38.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:39.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:40.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:41.107 7f0b59073a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:41.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:41.969 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:38:41.970 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:38:41.970 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:38:41.970 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:38:41.970 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:38:41.970 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:38:41.970 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:38:41.970 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:38:42.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:43.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:44.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:46.109 7f0b59073a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:46.109 7f0b59073a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:46.110 7f0b59073a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:46.110 7f0b59073a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:46.110 7f0b59073a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:46.110 7f0b59073a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:46.110 7f0b59073a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:46.110 7f0b59073a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:46.110 7f0b59073a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:46.111 7f0b59073a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:46.111 7f0b59073a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:46.111 7f0b59073a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:38:46.111 7f0b59073a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:46.757 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:47.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:48.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:49.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:50.108 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:50.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:51.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:52.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:53.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:54.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:55.111 7f0b5908c198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:38:55.111 7f0b5908c198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:38:55.111 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:55.111 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:38:55.111 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:38:55.111 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:38:55.112 7f0b5908c198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:38:55.112 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:55.112 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:38:55.112 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:55.112 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:55.112 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:38:55.112 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:38:55.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:56.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:57.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:58.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:38:59.112 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:38:59.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:00.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:01.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:02.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:03.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:04.113 7f0b5908c198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:04.113 7f0b5908c198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:04.114 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:04.114 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:04.114 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:04.114 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:04.114 7f0b5908c198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:04.114 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:04.115 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:04.115 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:04.115 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:04.115 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:39:04.115 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:04.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:05.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:06.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:07.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:08.114 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:08.357 7f0b5907a2e8 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:39:08.412 7f0b5907a2e8 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:39:08.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:09.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:10.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:11.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:12.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:13.116 7f0b5908c198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:13.116 7f0b5908c198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:13.116 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:13.116 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:13.116 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:13.116 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:13.117 7f0b5908c198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:13.117 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:13.117 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:13.117 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:13.117 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:13.117 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:39:13.117 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:13.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:14.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:16.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:17.005 7f0b5908c198 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:39:17.056 7f0b5908c198 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:39:17.116 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:17.760 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:18.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:19.762 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:20.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:21.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:22.118 7f0b5908c198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:22.119 7f0b5908c198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:22.119 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:22.119 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:22.119 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:22.119 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:22.119 7f0b5908c198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:22.119 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:22.120 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:22.120 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:22.120 7f0b5908c198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:22.120 7f0b5908c198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:39:22.120 7f0b5908c198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:22.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:23.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:24.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:25.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:26.119 7f0b5914da20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:26.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:27.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:28.293 7f0b59073780 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:39:28.345 7f0b590a2ac8 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:39:28.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:29.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:30.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:31.122 7f0b5914da20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:31.122 7f0b5914da20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:31.122 7f0b5914da20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:31.122 7f0b5914da20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:31.122 7f0b5914da20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:31.122 7f0b5914da20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:31.123 7f0b5914da20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:31.123 7f0b5914da20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:31.123 7f0b5914da20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:31.123 7f0b5914da20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:31.123 7f0b5914da20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:31.123 7f0b5914da20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:39:31.123 7f0b5914da20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:31.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:32.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:33.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:34.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:35.122 7f0b590a2ac8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:35.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:36.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:37.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:38.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:39.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:40.125 7f0b590a2ac8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:40.125 7f0b590a2ac8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:40.125 7f0b590a2ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:40.125 7f0b590a2ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:40.125 7f0b590a2ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:40.125 7f0b590a2ac8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:40.126 7f0b590a2ac8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:40.126 7f0b590a2ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:40.126 7f0b590a2ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:40.126 7f0b590a2ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:40.126 7f0b590a2ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:40.126 7f0b590a2ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:39:40.126 7f0b590a2ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:40.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:41.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:42.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:42.864 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:39:42.864 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:39:42.864 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:39:42.864 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:39:42.864 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:39:42.865 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:39:42.865 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:39:42.865 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:39:43.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:44.125 7f0b59132828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:44.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:46.759 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:47.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:48.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:49.127 7f0b59132828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:49.127 7f0b59132828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:49.127 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:49.127 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:49.128 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:49.128 7f0b59132828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:49.128 7f0b59132828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:49.128 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:49.129 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:49.129 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:49.129 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:49.129 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:39:49.129 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:49.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:50.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:51.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:52.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:53.127 7f0b59132828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:39:53.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:54.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:55.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:56.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:57.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:58.130 7f0b59132828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:39:58.130 7f0b59132828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:39:58.130 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:58.130 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:39:58.130 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:39:58.130 7f0b59132828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:39:58.131 7f0b59132828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:39:58.131 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:58.131 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:39:58.131 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:58.131 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:58.131 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:39:58.131 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:39:58.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:39:59.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:00.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:01.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:02.130 7f0b59132828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:02.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:03.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:04.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:05.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:06.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:07.132 7f0b59132828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:07.132 7f0b59132828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:07.132 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:07.132 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:07.132 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:07.133 7f0b59132828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:07.133 7f0b59132828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:07.133 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:07.133 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:07.134 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:07.134 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:07.134 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:40:07.134 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:07.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:08.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:09.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:10.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:11.133 7f0b59132828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:11.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:12.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:13.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:14.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:16.135 7f0b59132828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:16.135 7f0b59132828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:16.135 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:16.136 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:16.136 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:16.136 7f0b59132828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:16.136 7f0b59132828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:16.136 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:16.136 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:16.137 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:16.137 7f0b59132828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:16.137 7f0b59132828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:40:16.137 7f0b59132828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:16.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:17.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:18.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:19.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:20.136 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:20.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:21.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:22.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:23.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:24.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:25.138 7f0b591384e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:25.139 7f0b591384e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:25.139 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:25.139 7f0b591384e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:25.139 7f0b591384e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:25.139 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:25.140 7f0b591384e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:25.140 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:25.140 7f0b591384e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:25.140 7f0b591384e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:25.140 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:25.140 7f0b591384e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:40:25.140 7f0b591384e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:25.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:26.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:27.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:28.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:29.139 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:29.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:30.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:31.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:32.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:33.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:34.142 7f0b591384e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:34.142 7f0b591384e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:34.142 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:34.142 7f0b591384e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:34.142 7f0b591384e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:34.142 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:34.143 7f0b591384e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:34.143 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:34.143 7f0b591384e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:34.143 7f0b591384e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:34.143 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:34.143 7f0b591384e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:40:34.143 7f0b591384e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:34.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:35.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:36.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:37.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:38.142 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:38.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:39.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:40.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:41.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:42.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:43.145 7f0b591384e0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:43.145 7f0b591384e0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:43.145 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:43.145 7f0b591384e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:43.145 7f0b591384e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:43.145 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:43.146 7f0b591384e0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:43.146 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:43.146 7f0b591384e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:43.146 7f0b591384e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:43.146 7f0b591384e0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:43.146 7f0b591384e0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:40:43.146 7f0b591384e0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:43.773 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:40:43.773 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:40:43.773 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:40:43.773 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:40:43.773 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:40:43.773 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:40:43.774 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:40:43.774 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:40:43.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:44.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:46.761 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:47.145 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:47.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:48.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:49.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:50.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:51.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:52.147 7f0b5911a198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:40:52.147 7f0b5911a198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:40:52.147 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:52.148 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:40:52.148 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:40:52.148 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:40:52.148 7f0b5911a198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:40:52.148 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:52.148 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:40:52.148 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:52.148 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:52.149 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:40:52.149 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:40:52.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:53.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:54.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:55.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:56.148 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:40:56.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:57.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:58.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:40:59.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:00.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:01.150 7f0b5911a198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:01.151 7f0b5911a198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:01.151 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:01.151 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:01.151 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:01.151 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:01.152 7f0b5911a198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:01.152 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:01.152 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:01.152 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:01.152 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:01.152 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:41:01.152 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:01.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:02.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:03.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:04.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:05.151 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:05.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:06.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:07.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:08.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:09.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:10.153 7f0b5911a198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:10.153 7f0b5911a198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:10.154 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:10.154 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:10.154 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:10.154 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:10.154 7f0b5911a198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:10.154 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:10.155 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:10.155 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:10.155 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:10.155 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:41:10.155 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:10.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:11.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:12.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:13.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:14.154 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:14.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:16.763 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:17.765 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:18.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:19.156 7f0b5911a198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:19.156 7f0b5911a198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:19.156 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:19.157 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:19.157 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:19.157 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:19.157 7f0b5911a198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:19.157 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:19.157 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:19.157 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:19.158 7f0b5911a198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:19.158 7f0b5911a198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:41:19.158 7f0b5911a198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:19.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:20.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:21.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:22.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:23.156 7f0b5936cd68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:23.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:24.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:25.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:26.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:27.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:28.159 7f0b5936cd68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:28.159 7f0b5936cd68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:28.159 7f0b5936cd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:28.159 7f0b5936cd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:28.159 7f0b5936cd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:28.159 7f0b5936cd68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:28.160 7f0b5936cd68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:28.160 7f0b5936cd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:28.160 7f0b5936cd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:28.160 7f0b5936cd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:28.160 7f0b5936cd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:28.160 7f0b5936cd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:41:28.160 7f0b5936cd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:28.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:29.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:30.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:31.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:32.160 7f0b5936cd68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:32.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:33.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:34.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:35.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:36.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:37.162 7f0b5936cd68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:37.162 7f0b5936cd68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:37.162 7f0b5936cd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:37.162 7f0b5936cd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:37.162 7f0b5936cd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:37.162 7f0b5936cd68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:37.163 7f0b5936cd68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:37.163 7f0b5936cd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:37.163 7f0b5936cd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:37.163 7f0b5936cd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:37.163 7f0b5936cd68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:37.163 7f0b5936cd68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:41:37.163 7f0b5936cd68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:37.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:38.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:39.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:40.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:41.162 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:41.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:42.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:43.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:44.673 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:41:44.673 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:41:44.673 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:41:44.673 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:41:44.673 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:41:44.673 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:41:44.673 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:41:44.674 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:41:44.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:46.165 7f0b5909f198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:46.165 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:46.165 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:46.165 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:46.165 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:46.166 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:46.166 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:46.166 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:46.166 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:46.166 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:46.166 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:46.167 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:41:46.167 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:46.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:47.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:48.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:49.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:50.166 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:50.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:51.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:52.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:53.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:54.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:55.167 7f0b5909f198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:41:55.168 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:41:55.168 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:55.168 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:41:55.168 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:41:55.168 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:41:55.168 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:41:55.169 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:55.169 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:41:55.169 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:55.169 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:55.169 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:41:55.169 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:41:55.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:56.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:57.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:58.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:41:59.168 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:41:59.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:00.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:01.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:02.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:03.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:04.170 7f0b5909f198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:04.171 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:04.171 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:04.171 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:04.171 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:04.171 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:04.171 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:04.172 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:04.172 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:04.172 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:04.172 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:04.172 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:42:04.172 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:04.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:05.461 7f0b5909f198 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:42:05.513 7f0b5909f198 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:42:05.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:06.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:07.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:08.171 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:08.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:09.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:10.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:11.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:12.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:13.173 7f0b5909f198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:13.173 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:13.173 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:13.173 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:13.173 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:13.173 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:13.174 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:13.174 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:13.174 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:13.174 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:13.174 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:13.174 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:42:13.174 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:13.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:14.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:16.764 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:17.174 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:17.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:18.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:19.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:20.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:21.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:22.176 7f0b5909f198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:22.176 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:22.176 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:22.177 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:22.177 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:22.177 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:22.177 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:22.177 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:22.178 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:22.178 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:22.178 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:22.178 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:42:22.178 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:22.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:23.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:24.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:25.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:26.177 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:26.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:27.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:28.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:29.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:30.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:31.179 7f0b5909f198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:31.179 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:31.179 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:31.179 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:31.179 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:31.179 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:31.180 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:31.180 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:31.180 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:31.180 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:31.180 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:31.180 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:42:31.180 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:31.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:32.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:33.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:34.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:35.180 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:35.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:36.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:37.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:38.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:39.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:40.182 7f0b5909f198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:40.182 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:40.182 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:40.183 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:40.183 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:40.183 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:40.183 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:40.184 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:40.184 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:40.184 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:40.184 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:40.184 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:42:40.184 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:40.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:41.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:42.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:43.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:44.183 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:44.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:45.575 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:42:45.575 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:42:45.575 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:42:45.576 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:42:45.576 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:42:45.576 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:42:45.576 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:42:45.576 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:42:46.766 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:47.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:48.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:49.185 7f0b5909f198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:49.185 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:49.186 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:49.186 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:49.186 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:49.186 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:49.186 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:49.186 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:49.187 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:49.187 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:49.187 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:49.187 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:42:49.187 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:49.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:50.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:51.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:52.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:53.186 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:42:53.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:54.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:55.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:56.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:57.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:58.189 7f0b5909f198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:42:58.189 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:42:58.189 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:58.189 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:42:58.189 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:42:58.189 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:42:58.190 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:42:58.190 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:58.190 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:42:58.190 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:58.190 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:58.190 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:42:58.190 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:42:58.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:42:59.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:00.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:01.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:02.190 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:02.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:03.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:04.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:05.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:06.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:07.192 7f0b5909f198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:07.192 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:07.192 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:07.192 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:07.192 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:07.192 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:07.193 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:07.193 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:07.193 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:07.193 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:07.193 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:07.194 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:43:07.194 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:07.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:08.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:09.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:10.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:11.193 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:11.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:12.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:13.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:14.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:16.195 7f0b5909f198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:16.195 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:16.195 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:16.196 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:16.196 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:16.196 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:16.196 7f0b5909f198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:16.196 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:16.196 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:16.197 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:16.197 7f0b5909f198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:16.197 7f0b5909f198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:43:16.197 7f0b5909f198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:16.767 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:17.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:18.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:19.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:20.196 7f0b59365048 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:20.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:21.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:22.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:23.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:24.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:25.198 7f0b59365048 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:25.199 7f0b59365048 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:25.199 7f0b59365048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:25.199 7f0b59365048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:25.199 7f0b59365048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:25.199 7f0b59365048 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:25.199 7f0b59365048 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:25.199 7f0b59365048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:25.200 7f0b59365048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:25.200 7f0b59365048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:25.200 7f0b59365048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:25.200 7f0b59365048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:43:25.200 7f0b59365048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:25.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:26.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:27.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:28.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:29.199 7f0b59365048 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:29.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:30.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:31.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:32.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:33.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:34.201 7f0b59365048 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:34.201 7f0b59365048 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:34.201 7f0b59365048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:34.202 7f0b59365048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:34.202 7f0b59365048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:34.202 7f0b59365048 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:34.202 7f0b59365048 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:34.202 7f0b59365048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:34.202 7f0b59365048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:34.203 7f0b59365048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:34.203 7f0b59365048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:34.203 7f0b59365048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:43:34.203 7f0b59365048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:34.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:35.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:36.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:37.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:38.202 7f0b59365048 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:38.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:39.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:40.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:41.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:42.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:43.204 7f0b59365048 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:43.204 7f0b59365048 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:43.204 7f0b59365048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:43.204 7f0b59365048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:43.204 7f0b59365048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:43.204 7f0b59365048 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:43.205 7f0b59365048 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:43.205 7f0b59365048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:43.205 7f0b59365048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:43.205 7f0b59365048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:43.205 7f0b59365048 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:43.205 7f0b59365048 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:43:43.205 7f0b59365048 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:43.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:44.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:46.464 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:43:46.464 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:43:46.464 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:43:46.464 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:43:46.464 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:43:46.464 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:43:46.465 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:43:46.465 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:43:46.768 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:47.204 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:47.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:48.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:49.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:50.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:51.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:52.207 7f0b5907a588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:43:52.207 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:43:52.207 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:52.207 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:43:52.207 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:43:52.207 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:43:52.208 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:43:52.208 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:52.209 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:43:52.209 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:52.209 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:52.209 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:43:52.209 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:43:52.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:53.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:54.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:55.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:56.207 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:43:56.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:57.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:58.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:43:59.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:00.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:01.209 7f0b5907a588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:01.210 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:01.210 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:01.210 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:01.210 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:01.210 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:01.210 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:01.211 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:01.211 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:01.211 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:01.211 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:01.211 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:44:01.211 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:01.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:02.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:03.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:04.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:05.210 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:05.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:06.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:07.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:08.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:09.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:10.212 7f0b5907a588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:10.212 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:10.212 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:10.213 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:10.213 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:10.213 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:10.213 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:10.213 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:10.213 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:10.214 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:10.214 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:10.214 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:44:10.214 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:10.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:11.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:12.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:13.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:14.213 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:14.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:15.313 7f0b590ad048 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:44:15.364 7f0b590ad048 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:44:16.769 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:17.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:18.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:19.215 7f0b5907a588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:19.215 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:19.215 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:19.215 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:19.215 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:19.215 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:19.216 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:19.216 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:19.216 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:19.216 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:19.216 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:19.216 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:44:19.216 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:19.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:20.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:21.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:22.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:23.215 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:23.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:23.962 7f0b595a40f0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:44:24.012 7f0b595a40f0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:44:24.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:25.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:26.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:27.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:28.218 7f0b5907a588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:28.218 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:28.218 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:28.218 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:28.218 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:28.218 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:28.219 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:28.219 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:28.219 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:28.219 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:28.219 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:28.219 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:44:28.220 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:28.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:29.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:30.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:31.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:32.218 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:32.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:33.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:34.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:35.824 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:36.825 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:37.222 7f0b5907a588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:37.222 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:37.222 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:37.222 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:37.222 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:37.222 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:37.223 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:37.223 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:37.223 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:37.223 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:37.223 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:37.223 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:44:37.223 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:37.827 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:38.162 7f0b5907a588 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:44:38.214 7f0b5907a588 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:44:38.828 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:39.829 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:40.830 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:41.222 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:41.832 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:42.833 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:43.834 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:44.836 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:46.224 7f0b5907a588 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:46.224 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:46.224 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:46.224 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:46.224 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:46.225 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:46.225 7f0b5907a588 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:46.225 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:46.225 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:46.225 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:46.225 7f0b5907a588 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:46.226 7f0b5907a588 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:44:46.226 7f0b5907a588 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:46.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:47.400 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:44:47.400 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:44:47.400 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:44:47.400 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:44:47.400 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:44:47.400 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:44:47.400 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:44:47.401 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:44:47.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:48.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:49.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:50.224 7f0b59365978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:50.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:51.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:52.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:53.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:54.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:55.227 7f0b59365978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:44:55.227 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:44:55.227 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:55.228 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:44:55.228 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:44:55.228 7f0b59365978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:44:55.228 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:44:55.228 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:55.228 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:44:55.228 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:55.228 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:55.229 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:44:55.229 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:44:55.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:56.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:57.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:58.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:44:59.227 7f0b59365978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:44:59.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:00.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:01.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:02.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:03.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:04.229 7f0b59365978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:04.229 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:04.229 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:04.230 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:04.230 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:04.230 7f0b59365978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:04.230 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:04.230 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:04.230 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:04.230 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:04.230 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:04.231 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:45:04.231 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:04.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:05.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:06.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:07.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:08.230 7f0b59365978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:08.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:09.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:10.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:11.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:12.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:13.232 7f0b59365978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:13.232 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:13.232 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:13.233 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:13.233 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:13.233 7f0b59365978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:13.233 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:13.233 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:13.234 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:13.234 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:13.234 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:13.234 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:45:13.234 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:13.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:14.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:16.770 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:17.233 7f0b59365978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:17.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:18.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:19.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:20.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:21.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:22.234 7f0b59365978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:22.235 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:22.235 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:22.235 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:22.235 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:22.235 7f0b59365978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:22.236 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:22.236 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:22.236 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:22.236 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:22.236 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:22.236 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:45:22.236 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:22.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:23.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:24.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:25.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:26.235 7f0b59365978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:26.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:27.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:28.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:29.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:30.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:31.238 7f0b59365978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:31.238 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:31.238 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:31.238 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:31.238 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:31.238 7f0b59365978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:31.239 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:31.239 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:31.239 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:31.239 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:31.239 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:31.239 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:45:31.239 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:31.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:32.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:33.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:34.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:35.238 7f0b59365978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:35.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:36.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:37.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:38.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:39.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:40.240 7f0b59365978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:40.240 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:40.240 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:40.241 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:40.241 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:40.241 7f0b59365978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:40.241 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:40.241 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:40.242 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:40.242 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:40.242 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:40.242 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:45:40.242 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:40.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:41.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:42.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:43.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:44.241 7f0b59365978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:44.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:46.771 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:47.772 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:48.293 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:45:48.293 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:45:48.294 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:45:48.294 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:45:48.294 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:45:48.294 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:45:48.294 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:45:48.294 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:45:48.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:49.243 7f0b59365978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:49.243 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:49.243 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:49.243 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:49.243 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:49.243 7f0b59365978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:49.244 7f0b59365978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:49.244 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:49.244 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:49.244 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:49.244 7f0b59365978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:49.244 7f0b59365978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:45:49.244 7f0b59365978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:49.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:50.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:51.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:52.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:53.243 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:45:53.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:54.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:55.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:56.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:57.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:58.245 7f0b5a804b70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:45:58.245 7f0b5a804b70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:45:58.245 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:58.245 7f0b5a804b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:45:58.245 7f0b5a804b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:45:58.245 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:45:58.246 7f0b5a804b70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:45:58.246 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:58.246 7f0b5a804b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:45:58.246 7f0b5a804b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:58.246 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:58.246 7f0b5a804b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:45:58.246 7f0b5a804b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:45:58.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:45:59.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:00.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:01.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:02.245 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:02.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:03.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:04.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:05.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:06.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:07.247 7f0b5a804b70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:07.248 7f0b5a804b70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:07.248 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:07.248 7f0b5a804b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:07.248 7f0b5a804b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:07.248 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:07.249 7f0b5a804b70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:07.249 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:07.249 7f0b5a804b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:07.249 7f0b5a804b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:07.249 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:07.249 7f0b5a804b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:46:07.249 7f0b5a804b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:07.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:08.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:09.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:10.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:11.248 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:11.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:12.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:13.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:14.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:16.251 7f0b5a804b70 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:16.251 7f0b5a804b70 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:16.252 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:16.252 7f0b5a804b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:16.252 7f0b5a804b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:16.252 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:16.252 7f0b5a804b70 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:16.252 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:16.253 7f0b5a804b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:16.253 7f0b5a804b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:16.253 7f0b5a804b70 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:16.253 7f0b5a804b70 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:46:16.253 7f0b5a804b70 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:16.773 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:17.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:18.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:19.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:20.254 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:20.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:21.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:22.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:23.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:24.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:25.257 7f0b59405a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:25.258 7f0b59405a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:25.258 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:25.258 7f0b59405a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:25.258 7f0b59405a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:25.258 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:25.259 7f0b59405a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:25.259 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:25.259 7f0b59405a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:25.259 7f0b59405a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:25.259 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:25.259 7f0b59405a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:46:25.259 7f0b59405a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:25.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:26.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:27.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:28.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:29.258 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:29.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:30.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:31.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:32.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:33.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:34.261 7f0b59405a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:34.261 7f0b59405a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:34.261 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:34.261 7f0b59405a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:34.261 7f0b59405a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:34.261 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:34.262 7f0b59405a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:34.262 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:34.262 7f0b59405a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:34.262 7f0b59405a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:34.262 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:34.262 7f0b59405a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:46:34.262 7f0b59405a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:34.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:35.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:36.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:37.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:38.262 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:38.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:39.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:40.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:41.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:42.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:43.264 7f0b59405a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:43.264 7f0b59405a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:43.264 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:43.264 7f0b59405a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:43.264 7f0b59405a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:43.265 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:43.265 7f0b59405a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:43.265 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:43.265 7f0b59405a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:43.265 7f0b59405a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:43.265 7f0b59405a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:43.265 7f0b59405a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:46:43.266 7f0b59405a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:43.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:44.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:46.774 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:47.177 7f0b59405a20 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:46:47.228 7f0b59405a20 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:46:47.264 7f0b59067828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:47.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:48.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:49.178 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:46:49.178 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:46:49.178 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:46:49.178 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:46:49.178 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:46:49.178 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:46:49.178 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:46:49.178 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:46:49.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:50.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:51.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:52.266 7f0b59067828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:46:52.266 7f0b59067828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:46:52.267 7f0b59067828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:52.267 7f0b59067828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:46:52.267 7f0b59067828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:46:52.267 7f0b59067828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:46:52.267 7f0b59067828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:46:52.268 7f0b59067828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:52.268 7f0b59067828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:46:52.268 7f0b59067828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:52.268 7f0b59067828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:52.268 7f0b59067828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:46:52.268 7f0b59067828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:46:52.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:53.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:54.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:55.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:56.267 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:46:56.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:57.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:58.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:46:59.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:00.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:01.268 7f0b58efef60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:01.268 7f0b58efef60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:01.269 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:01.269 7f0b58efef60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:01.269 7f0b58efef60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:01.269 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:01.269 7f0b58efef60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:01.269 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:01.270 7f0b58efef60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:01.270 7f0b58efef60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:01.270 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:01.270 7f0b58efef60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:47:01.270 7f0b58efef60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:01.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:02.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:03.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:04.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:05.270 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:05.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:06.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:07.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:08.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:09.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:10.272 7f0b58efef60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:10.272 7f0b58efef60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:10.272 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:10.272 7f0b58efef60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:10.273 7f0b58efef60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:10.273 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:10.273 7f0b58efef60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:10.273 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:10.273 7f0b58efef60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:10.273 7f0b58efef60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:10.273 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:10.274 7f0b58efef60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:47:10.274 7f0b58efef60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:10.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:11.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:12.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:13.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:14.273 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:14.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:16.775 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:17.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:18.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:19.275 7f0b58efef60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:19.275 7f0b58efef60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:19.275 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:19.276 7f0b58efef60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:19.276 7f0b58efef60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:19.276 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:19.276 7f0b58efef60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:19.276 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:19.276 7f0b58efef60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:19.276 7f0b58efef60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:19.277 7f0b58efef60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:19.277 7f0b58efef60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:47:19.277 7f0b58efef60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:19.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:20.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:21.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:22.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:23.276 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:23.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:24.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:25.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:26.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:27.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:28.282 7f0b58ef0ac8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:28.282 7f0b58ef0ac8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:28.282 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:28.282 7f0b58ef0ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:28.282 7f0b58ef0ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:28.282 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:28.283 7f0b58ef0ac8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:28.283 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:28.283 7f0b58ef0ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:28.283 7f0b58ef0ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:28.283 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:28.283 7f0b58ef0ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:47:28.283 7f0b58ef0ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:28.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:29.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:30.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:31.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:32.282 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:32.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:33.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:34.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:35.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:36.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:37.284 7f0b58ef0ac8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:37.285 7f0b58ef0ac8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:37.285 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:37.285 7f0b58ef0ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:37.285 7f0b58ef0ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:37.285 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:37.285 7f0b58ef0ac8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:37.285 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:37.286 7f0b58ef0ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:37.286 7f0b58ef0ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:37.286 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:37.286 7f0b58ef0ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:47:37.286 7f0b58ef0ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:37.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:38.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:39.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:40.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:41.285 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:41.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:42.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:43.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:44.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:46.288 7f0b58ef0ac8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:46.288 7f0b58ef0ac8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:46.288 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:46.289 7f0b58ef0ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:46.289 7f0b58ef0ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:46.289 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:46.289 7f0b58ef0ac8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:46.289 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:46.289 7f0b58ef0ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:46.290 7f0b58ef0ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:46.290 7f0b58ef0ac8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:46.290 7f0b58ef0ac8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:47:46.290 7f0b58ef0ac8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:46.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:47.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:48.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:49.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:50.064 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:47:50.064 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:47:50.064 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:47:50.064 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:47:50.064 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:47:50.065 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:47:50.065 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:47:50.065 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:47:50.289 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:50.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:51.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:52.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:53.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:54.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:55.291 7f0b58ee5d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:47:55.292 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:47:55.292 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:55.292 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:47:55.292 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:47:55.292 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:47:55.293 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:47:55.293 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:55.293 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:47:55.293 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:55.293 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:55.293 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:47:55.293 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:47:55.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:56.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:57.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:58.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:47:59.292 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:47:59.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:00.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:01.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:02.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:03.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:04.295 7f0b58ee5d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:04.295 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:04.295 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:04.295 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:04.295 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:04.296 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:04.296 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:04.296 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:04.296 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:04.296 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:04.297 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:04.297 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:48:04.297 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:04.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:05.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:06.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:07.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:08.295 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:08.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:09.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:10.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:11.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:12.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:13.298 7f0b58ee5d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:13.298 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:13.298 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:13.298 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:13.298 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:13.299 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:13.299 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:13.299 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:13.299 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:13.299 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:13.299 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:13.299 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:48:13.300 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:13.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:14.820 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:16.776 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:17.299 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:17.778 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:18.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:19.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:20.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:21.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:22.301 7f0b58ee5d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:22.301 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:22.301 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:22.302 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:22.302 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:22.302 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:22.302 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:22.302 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:22.302 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:22.302 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:22.302 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:22.303 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:48:22.303 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:22.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:23.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:24.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:25.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:26.301 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:26.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:27.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:28.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:29.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:30.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:31.304 7f0b58ee5d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:31.304 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:31.304 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:31.305 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:31.305 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:31.305 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:31.305 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:31.305 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:31.305 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:31.305 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:31.306 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:31.306 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:48:31.306 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:31.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:32.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:33.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:34.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:35.305 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:35.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:36.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:37.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:38.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:39.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:40.307 7f0b58ee5d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:40.307 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:40.307 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:40.308 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:40.308 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:40.308 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:40.308 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:40.308 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:40.308 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:40.308 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:40.308 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:40.309 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:48:40.309 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:40.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:41.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:42.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:43.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:44.308 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:44.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:46.777 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:47.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:48.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:49.311 7f0b58ee5d68 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:49.311 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:49.311 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:49.311 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:49.311 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:49.311 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:49.312 7f0b58ee5d68 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:49.312 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:49.312 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:49.312 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:49.312 7f0b58ee5d68 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:49.312 7f0b58ee5d68 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:48:49.312 7f0b58ee5d68 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:49.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:50.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:50.961 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:48:50.961 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:48:50.961 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:48:50.961 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:48:50.961 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:48:50.961 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:48:50.961 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:48:50.962 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:48:51.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:52.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:53.311 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:48:53.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:54.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:55.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:56.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:57.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:58.314 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:48:58.314 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:48:58.314 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:58.314 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:48:58.314 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:48:58.315 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:48:58.315 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:48:58.315 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:58.315 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:48:58.316 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:58.316 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:58.316 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:48:58.316 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:48:58.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:48:59.418 7f0b5936e828 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:48:59.469 7f0b5936e828 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:48:59.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:00.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:01.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:02.314 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:02.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:03.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:04.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:05.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:06.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:07.316 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:07.316 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:07.316 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:07.316 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:07.317 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:07.317 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:07.317 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:07.317 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:07.317 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:07.317 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:07.317 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:07.318 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:49:07.318 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:07.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:08.049 7f0b5936e828 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:49:08.105 7f0b5936e828 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:49:08.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:09.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:10.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:11.317 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:11.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:12.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:13.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:14.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:16.319 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:16.319 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:16.319 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:16.320 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:16.320 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:16.320 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:16.321 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:16.321 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:16.321 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:16.321 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:16.321 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:16.321 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:49:16.321 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:16.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:17.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:18.781 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:19.269 7f0b5936e828 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:49:19.320 7f0b5936e828 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:49:19.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:20.320 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:20.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:21.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:22.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:23.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:24.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:25.322 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:25.322 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:25.322 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:25.322 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:25.322 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:25.322 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:25.323 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:25.323 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:25.323 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:25.323 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:25.323 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:25.323 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:49:25.323 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:25.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:26.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:27.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:28.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:29.322 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:29.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:30.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:31.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:32.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:33.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:34.323 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:34.324 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:34.324 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:34.324 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:34.324 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:34.324 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:34.325 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:34.325 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:34.325 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:34.325 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:34.325 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:34.325 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:49:34.325 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:34.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:35.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:36.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:37.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:38.324 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:38.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:39.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:40.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:41.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:42.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:43.326 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:43.327 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:43.327 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:43.327 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:43.327 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:43.327 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:43.328 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:43.328 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:43.328 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:43.328 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:43.328 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:43.328 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:49:43.328 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:43.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:44.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:46.779 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:47.327 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:47.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:48.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:49.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:50.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:51.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:51.868 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:49:51.868 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:49:51.868 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:49:51.868 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:49:51.868 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:49:51.869 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:49:51.869 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:49:51.869 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:49:52.330 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:49:52.330 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:49:52.330 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:52.330 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:49:52.330 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:49:52.330 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:49:52.331 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:49:52.331 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:52.331 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:49:52.331 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:52.331 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:52.331 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:49:52.331 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:49:52.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:53.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:54.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:55.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:56.331 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:49:56.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:57.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:58.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:49:59.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:00.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:01.333 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:01.333 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:01.333 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:01.334 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:01.334 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:01.334 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:01.334 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:01.335 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:01.335 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:01.335 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:01.335 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:01.335 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:01.335 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:01.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:02.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:03.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:04.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:05.334 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:05.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:06.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:07.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:08.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:09.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:10.336 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:10.336 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:10.336 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:10.337 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:10.337 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:10.337 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:10.337 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:10.337 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:10.337 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:10.337 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:10.338 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:10.338 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:10.338 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:10.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:11.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:12.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:13.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:14.337 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:14.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:16.780 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:17.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:18.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:19.342 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:19.342 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:19.342 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:19.342 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:19.342 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:19.342 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:19.343 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:19.343 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:19.343 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:19.343 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:19.343 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:19.343 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:19.343 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:19.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:20.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:21.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:22.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:23.350 7f0b5932e978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:23.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:24.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:25.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:26.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:27.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:28.353 7f0b5932e978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:28.353 7f0b5932e978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:28.353 7f0b5932e978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:28.353 7f0b5932e978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:28.353 7f0b5932e978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:28.353 7f0b5932e978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:28.354 7f0b5932e978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:28.354 7f0b5932e978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:28.354 7f0b5932e978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:28.354 7f0b5932e978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:28.354 7f0b5932e978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:28.354 7f0b5932e978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:28.354 7f0b5932e978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:28.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:29.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:30.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:31.820 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:32.353 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:32.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:33.823 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:34.825 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:35.826 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:36.828 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:37.355 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:37.355 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:37.355 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:37.355 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:37.355 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:37.355 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:37.356 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:37.356 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:37.356 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:37.356 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:37.356 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:37.356 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:37.356 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:37.829 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:38.831 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:39.833 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:40.834 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:41.355 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:41.837 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:42.838 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:43.840 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:44.842 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:46.357 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:46.357 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:46.358 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:46.358 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:46.358 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:46.358 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:46.358 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:46.358 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:46.359 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:46.359 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:46.359 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:46.359 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:46.359 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:46.782 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:47.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:48.237 7f0b5936e828 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:50:48.289 7f0b5936e828 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:50:48.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:49.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:50.358 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:50.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:51.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:52.758 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:50:52.758 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:50:52.758 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:50:52.758 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:50:52.758 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:50:52.758 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:50:52.758 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:50:52.759 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:50:52.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:53.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:54.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:55.360 7f0b5936e828 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:50:55.360 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:50:55.361 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:55.361 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:50:55.361 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:50:55.361 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:50:55.361 7f0b5936e828 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:50:55.361 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:55.362 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:50:55.362 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:55.362 7f0b5936e828 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:55.362 7f0b5936e828 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:50:55.362 7f0b5936e828 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:50:55.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:56.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:57.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:58.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:50:59.361 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:50:59.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:00.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:01.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:02.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:03.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:04.363 7f0b58ee9a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:04.364 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:04.364 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:04.364 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:04.364 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:04.364 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:04.365 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:04.365 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:04.365 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:04.365 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:04.365 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:04.365 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:04.365 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:04.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:05.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:06.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:07.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:08.364 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:08.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:09.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:10.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:11.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:12.823 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:13.366 7f0b58ee9a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:13.367 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:13.367 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:13.367 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:13.367 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:13.367 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:13.368 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:13.368 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:13.368 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:13.368 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:13.368 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:13.368 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:13.368 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:13.824 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:14.826 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:16.783 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:17.367 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:17.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:18.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:19.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:20.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:21.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:22.370 7f0b58ee9a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:22.370 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:22.370 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:22.370 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:22.370 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:22.370 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:22.371 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:22.371 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:22.371 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:22.371 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:22.371 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:22.371 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:22.371 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:22.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:23.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:24.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:25.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:26.370 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:26.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:27.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:28.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:29.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:30.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:31.372 7f0b58ee9a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:31.373 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:31.373 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:31.373 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:31.373 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:31.373 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:31.374 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:31.374 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:31.374 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:31.374 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:31.374 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:31.374 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:31.374 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:31.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:32.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:33.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:34.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:35.373 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:35.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:36.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:37.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:38.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:39.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:40.375 7f0b58ee9a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:40.376 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:40.376 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:40.376 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:40.376 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:40.376 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:40.377 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:40.377 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:40.377 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:40.377 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:40.377 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:40.378 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:40.378 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:40.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:41.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:42.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:43.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:44.376 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:44.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:46.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:47.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:48.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:49.379 7f0b58ee9a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:49.379 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:49.380 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:49.380 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:49.380 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:49.380 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:49.380 7f0b58ee9a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:49.380 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:49.381 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:49.381 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:49.381 7f0b58ee9a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:49.381 7f0b58ee9a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:49.381 7f0b58ee9a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:49.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:50.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:51.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:52.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:53.380 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:51:53.663 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:51:53.663 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:51:53.663 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:51:53.663 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:51:53.664 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:51:53.664 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:51:53.664 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:51:53.664 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:51:53.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:54.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:55.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:56.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:57.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:58.382 7f0b5908d978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:51:58.382 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:51:58.382 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:58.382 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:51:58.382 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:51:58.382 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:51:58.383 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:51:58.383 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:58.383 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:51:58.383 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:58.383 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:58.383 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:51:58.383 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:51:58.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:51:59.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:00.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:01.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:02.383 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:02.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:03.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:04.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:05.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:06.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:07.385 7f0b5908d978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:07.385 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:07.385 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:07.385 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:07.385 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:07.386 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:07.386 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:07.386 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:07.386 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:07.386 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:07.387 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:07.387 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:07.387 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:07.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:08.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:09.820 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:10.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:11.385 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:11.823 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:12.825 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:13.826 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:14.828 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:16.388 7f0b5908d978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:16.388 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:16.388 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:16.388 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:16.388 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:16.388 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:16.389 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:16.389 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:16.389 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:16.389 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:16.389 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:16.389 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:16.389 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:16.784 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:17.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:18.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:19.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:20.388 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:20.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:21.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:22.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:23.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:24.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:25.391 7f0b5908d978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:25.391 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:25.391 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:25.391 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:25.391 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:25.391 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:25.392 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:25.392 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:25.392 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:25.392 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:25.392 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:25.392 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:25.392 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:25.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:26.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:27.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:28.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:29.392 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:29.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:30.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:31.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:32.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:33.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:34.394 7f0b5908d978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:34.395 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:34.395 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:34.395 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:34.395 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:34.395 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:34.396 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:34.396 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:34.396 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:34.396 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:34.396 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:34.396 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:34.396 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:34.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:35.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:36.820 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:37.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:38.395 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:38.823 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:39.824 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:40.825 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:41.827 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:42.828 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:43.398 7f0b5908d978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:43.399 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:43.399 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:43.399 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:43.399 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:43.399 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:43.400 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:43.400 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:43.400 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:43.400 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:43.400 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:43.400 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:43.400 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:43.830 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:44.831 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:46.785 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:47.399 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:47.786 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:48.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:49.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:50.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:51.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:52.403 7f0b5908d978 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:52:52.403 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:52:52.403 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:52.403 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:52:52.403 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:52:52.403 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:52:52.404 7f0b5908d978 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:52:52.404 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:52.404 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:52:52.404 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:52.404 7f0b5908d978 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:52.404 7f0b5908d978 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:52:52.404 7f0b5908d978 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:52:52.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:53.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:54.549 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:52:54.549 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:52:54.549 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:52:54.549 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:52:54.549 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:52:54.550 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:52:54.550 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:52:54.550 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:52:54.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:55.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:56.403 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:52:56.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:57.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:58.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:52:59.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:00.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:01.406 7f0b590b7f60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:01.406 7f0b590b7f60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:01.407 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:01.407 7f0b590b7f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:01.407 7f0b590b7f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:01.407 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:01.408 7f0b590b7f60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:01.408 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:01.408 7f0b590b7f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:01.408 7f0b590b7f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:01.408 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:01.408 7f0b590b7f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:01.408 7f0b590b7f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:01.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:02.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:03.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:04.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:05.407 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:05.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:06.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:07.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:08.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:09.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:10.409 7f0b590b7f60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:10.409 7f0b590b7f60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:10.409 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:10.410 7f0b590b7f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:10.410 7f0b590b7f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:10.410 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:10.410 7f0b590b7f60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:10.410 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:10.410 7f0b590b7f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:10.411 7f0b590b7f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:10.411 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:10.411 7f0b590b7f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:10.411 7f0b590b7f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:10.823 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:11.825 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:12.827 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:13.828 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:14.410 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:14.829 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:16.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:17.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:18.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:19.412 7f0b590b7f60 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:19.413 7f0b590b7f60 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:19.413 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:19.413 7f0b590b7f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:19.413 7f0b590b7f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:19.413 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:19.413 7f0b590b7f60 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:19.414 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:19.414 7f0b590b7f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:19.414 7f0b590b7f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:19.414 7f0b590b7f60 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:19.414 7f0b590b7f60 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:19.414 7f0b590b7f60 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:19.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:20.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:21.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:22.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:23.414 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:23.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:24.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:25.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:26.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:27.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:28.416 7f0b590bc6d8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:28.416 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:28.416 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:28.416 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:28.416 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:28.416 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:28.417 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:28.417 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:28.417 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:28.417 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:28.417 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:28.417 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:28.417 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:28.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:29.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:30.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:31.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:32.417 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:32.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:33.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:34.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:35.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:36.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:37.420 7f0b590bc6d8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:37.420 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:37.420 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:37.421 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:37.421 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:37.421 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:37.421 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:37.421 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:37.421 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:37.422 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:37.422 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:37.422 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:37.422 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:37.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:38.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:39.820 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:40.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:41.420 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:41.823 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:42.825 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:43.826 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:44.827 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:46.422 7f0b590bc6d8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:46.423 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:46.423 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:46.423 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:46.423 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:46.423 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:46.423 7f0b590bc6d8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:46.424 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:46.424 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:46.424 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:46.424 7f0b590bc6d8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:46.424 7f0b590bc6d8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:46.424 7f0b590bc6d8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:46.787 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:47.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:48.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:49.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:50.423 7f0b590c7198 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:50.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:51.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:52.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:53.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:54.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:55.425 7f0b590c7198 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:53:55.425 7f0b590c7198 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:53:55.425 7f0b590c7198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:55.425 7f0b590c7198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:53:55.425 7f0b590c7198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:53:55.425 7f0b590c7198 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:53:55.426 7f0b590c7198 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:53:55.426 7f0b590c7198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:55.426 7f0b590c7198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:53:55.426 7f0b590c7198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:55.426 7f0b590c7198 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:55.426 7f0b590c7198 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:53:55.426 7f0b590c7198 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:53:55.439 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:53:55.439 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:53:55.439 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:53:55.440 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:53:55.440 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:53:55.440 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:53:55.440 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:53:55.440 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:53:55.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:56.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:57.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:58.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:53:59.425 7f0b58f03438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:53:59.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:00.808 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:01.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:02.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:03.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:04.427 7f0b58f03438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:04.428 7f0b58f03438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:04.428 7f0b58f03438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:04.428 7f0b58f03438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:04.428 7f0b58f03438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:04.428 7f0b58f03438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:04.429 7f0b58f03438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:04.429 7f0b58f03438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:04.429 7f0b58f03438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:04.429 7f0b58f03438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:04.429 7f0b58f03438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:04.429 7f0b58f03438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:04.429 7f0b58f03438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:04.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:05.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:06.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:07.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:08.428 7f0b58f03438 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:08.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:09.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:10.824 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:11.826 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:12.374 7f0b59326ac8 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:54:12.424 7f0b59326ac8 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:54:12.827 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:13.430 7f0b58f03438 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:13.431 7f0b58f03438 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:13.431 7f0b58f03438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:13.431 7f0b58f03438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:13.431 7f0b58f03438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:13.431 7f0b58f03438 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:13.432 7f0b58f03438 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:13.432 7f0b58f03438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:13.432 7f0b58f03438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:13.432 7f0b58f03438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:13.432 7f0b58f03438 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:13.432 7f0b58f03438 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:13.432 7f0b58f03438 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:13.829 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:14.832 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:16.788 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:17.431 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:17.789 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:18.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:19.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:20.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:20.989 7f0b5908d438 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:54:21.041 7f0b5908d438 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:54:21.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:22.434 7f0b5a804a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:22.434 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:22.434 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:22.434 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:22.434 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:22.434 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:22.435 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:22.435 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:22.435 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:22.435 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:22.435 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:22.435 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:22.435 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:22.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:23.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:24.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:25.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:26.434 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:26.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:27.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:28.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:29.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:30.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:31.436 7f0b5a804a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:31.437 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:31.437 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:31.437 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:31.437 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:31.437 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:31.437 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:31.437 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:31.438 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:31.438 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:31.438 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:31.438 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:31.438 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:31.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:32.194 7f0b5a804a20 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:54:32.244 7f0b5a804a20 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:54:32.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:33.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:34.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:35.437 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:35.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:36.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:37.820 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:38.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:39.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:40.439 7f0b5a804a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:40.440 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:40.440 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:40.440 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:40.440 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:40.440 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:40.441 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:40.441 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:40.441 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:40.441 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:40.441 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:40.441 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:40.441 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:40.824 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:41.826 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:42.827 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:43.830 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:44.440 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:44.831 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:46.790 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:47.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:48.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:49.442 7f0b5a804a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:49.443 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:49.443 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:49.443 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:49.443 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:49.443 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:49.444 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:49.444 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:49.444 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:49.444 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:49.444 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:49.444 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:49.444 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:49.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:50.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:51.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:52.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:53.443 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:54:53.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:54.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:55.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:56.388 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:54:56.388 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:54:56.388 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:54:56.388 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:54:56.388 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:54:56.388 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:54:56.388 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:54:56.389 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:54:56.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:57.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:58.446 7f0b5a804a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:54:58.446 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:54:58.446 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:58.446 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:54:58.446 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:54:58.446 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:54:58.447 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:54:58.447 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:58.447 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:54:58.447 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:58.447 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:58.447 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:54:58.447 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:54:58.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:54:59.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:00.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:01.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:02.446 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:02.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:03.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:04.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:05.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:06.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:07.449 7f0b5a804a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:07.449 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:07.449 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:07.450 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:07.450 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:07.450 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:07.450 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:07.450 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:07.450 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:07.450 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:07.450 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:07.451 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:07.451 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:07.825 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:08.827 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:09.829 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:10.830 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:11.450 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:11.832 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:12.835 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:13.836 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:14.838 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:16.452 7f0b5a804a20 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:16.452 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:16.452 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:16.453 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:16.453 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:16.453 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:16.453 7f0b5a804a20 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:16.453 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:16.454 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:16.454 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:16.454 7f0b5a804a20 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:16.454 7f0b5a804a20 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:16.454 7f0b5a804a20 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:16.791 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:17.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:18.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:19.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:20.452 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:20.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:21.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:22.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:23.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:24.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:25.455 7f0b58eb9eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:25.455 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:25.455 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:25.455 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:25.455 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:25.456 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:25.456 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:25.456 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:25.456 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:25.456 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:25.456 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:25.456 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:25.456 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:25.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:26.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:27.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:28.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:29.456 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:29.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:30.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:31.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:32.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:33.820 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:34.458 7f0b58eb9eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:34.458 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:34.458 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:34.458 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:34.459 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:34.459 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:34.459 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:34.459 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:34.459 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:34.459 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:34.459 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:34.460 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:34.460 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:34.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:35.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:36.824 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:37.826 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:38.459 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:38.827 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:39.831 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:40.835 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:41.838 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:42.840 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:43.465 7f0b58eb9eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:43.465 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:43.465 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:43.465 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:43.466 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:43.466 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:43.466 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:43.466 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:43.467 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:43.467 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:43.467 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:43.467 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:43.467 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:43.841 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:44.843 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:46.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:47.466 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:47.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:48.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:49.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:50.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:51.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:52.468 7f0b58eb9eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:55:52.469 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:55:52.469 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:52.469 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:55:52.469 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:55:52.469 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:55:52.469 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:55:52.470 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:52.470 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:55:52.470 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:52.470 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:52.470 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:55:52.470 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:55:52.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:53.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:54.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:55.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:56.469 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:55:56.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:57.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:55:57.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:55:57.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:55:57.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:55:57.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:55:57.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:55:57.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:55:57.300 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:55:57.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:58.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:55:59.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:00.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:01.471 7f0b58eb9eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:01.472 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:01.472 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:01.472 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:01.472 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:01.472 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:01.473 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:01.473 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:01.473 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:01.473 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:01.473 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:01.473 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:01.473 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:01.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:02.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:03.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:04.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:05.473 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:05.820 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:06.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:07.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:08.824 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:09.825 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:10.475 7f0b58eb9eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:10.475 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:10.475 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:10.476 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:10.476 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:10.476 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:10.476 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:10.476 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:10.476 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:10.477 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:10.477 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:10.477 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:10.477 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:10.827 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:11.829 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:12.830 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:13.832 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:14.476 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:14.833 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:16.792 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:17.794 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:18.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:19.478 7f0b58eb9eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:19.479 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:19.479 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:19.479 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:19.479 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:19.479 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:19.480 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:19.480 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:19.480 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:19.480 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:19.480 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:19.480 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:19.480 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:19.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:20.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:21.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:22.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:23.480 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:23.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:24.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:25.805 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:26.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:27.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:28.482 7f0b58eb9eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:28.482 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:28.482 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:28.483 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:28.483 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:28.483 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:28.483 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:28.483 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:28.483 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:28.483 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:28.483 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:28.484 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:28.484 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:28.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:29.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:30.814 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:31.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:32.483 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:32.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:33.818 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:34.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:35.821 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:36.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:37.486 7f0b58eb9eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:37.486 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:37.486 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:37.486 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:37.486 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:37.486 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:37.487 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:37.487 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:37.487 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:37.487 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:37.487 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:37.487 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:37.487 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:37.616 7f0b58eb9eb8 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:56:37.668 7f0b58eb9eb8 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:56:37.823 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:38.825 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:39.826 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:40.828 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:41.487 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:41.830 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:42.831 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:43.833 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:44.834 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:46.489 7f0b58eb9eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:46.489 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:46.489 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:46.489 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:46.489 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:46.489 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:46.490 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:46.490 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:46.490 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:46.490 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:46.490 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:46.490 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:46.490 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:46.793 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:47.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:48.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:49.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:50.490 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:50.798 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:51.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:52.801 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:53.802 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:54.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:55.492 7f0b58eb9eb8 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:56:55.492 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:56:55.492 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:55.493 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:56:55.493 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:56:55.493 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:56:55.493 7f0b58eb9eb8 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:56:55.493 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:55.493 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:56:55.494 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:55.494 7f0b58eb9eb8 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:55.494 7f0b58eb9eb8 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:56:55.494 7f0b58eb9eb8 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:56:55.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:56.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:57.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:58.176 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_interfaces in resource virtual-router: 'virtual_machine_interfaces' 03/31/2025 02:56:58.176 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field network_ipam_refs in resource virtual-router: 'network_ipam_refs' 03/31/2025 02:56:58.176 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field sub_cluster_refs in resource virtual-router: 'sub_cluster_refs' 03/31/2025 02:56:58.176 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field virtual_machine_refs in resource virtual-router: 'virtual_machine_refs' 03/31/2025 02:56:58.176 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field tag_refs in resource virtual-router: 'tag_refs' 03/31/2025 02:56:58.177 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field instance_ip_back_refs in resource virtual-router: 'instance_ip_back_refs' 03/31/2025 02:56:58.177 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field physical_router_back_refs in resource virtual-router: 'physical_router_back_refs' 03/31/2025 02:56:58.177 7f0b595a24e0 [contrail-api] [WARNING]: __default__ [SYS_WARN]: VncApiDebug: Key not found for field provider_attachment_back_refs in resource virtual-router: 'provider_attachment_back_refs' 03/31/2025 02:56:58.811 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:56:59.493 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:56:59.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:00.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:01.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:02.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:03.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:04.496 7f0b58f140f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:04.496 7f0b58f140f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:04.496 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:04.496 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:04.496 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:04.496 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:04.497 7f0b58f140f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:04.497 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:04.497 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:04.497 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:04.497 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:04.497 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:57:04.497 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:04.820 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:05.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:06.823 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:07.825 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:08.496 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:08.827 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:09.828 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:10.830 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:11.832 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:12.834 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:13.498 7f0b58f140f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:13.498 7f0b58f140f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:13.499 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:13.499 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:13.499 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:13.499 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:13.500 7f0b58f140f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:13.500 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:13.500 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:13.500 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:13.500 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:13.500 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:57:13.500 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:13.835 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:14.837 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:16.795 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:17.499 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:17.796 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:18.797 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:19.799 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:20.800 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:21.803 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:22.502 7f0b58f140f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:22.502 7f0b58f140f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:22.502 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:22.502 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:22.502 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:22.503 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:22.503 7f0b58f140f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:22.503 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:22.503 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:22.503 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:22.503 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:22.503 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:57:22.503 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:22.804 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:23.806 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:24.807 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:25.809 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:26.503 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:26.810 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:27.812 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:28.813 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:29.815 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:30.816 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:31.505 7f0b58f140f0 [contrail-api] [ERROR]: Session Event: TCP Connect Fail 03/31/2025 02:57:31.505 7f0b58f140f0 [contrail-api] [INFO]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect] 03/31/2025 02:57:31.505 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:31.506 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Idle to Connect on EvIdleHoldTimerExpired] connection down >>, ] >> 03/31/2025 02:57:31.506 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Connect] 03/31/2025 02:57:31.506 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvTcpConnectFail] in state[Connect] 03/31/2025 02:57:31.506 7f0b58f140f0 [contrail-api] [INFO]: Sandesh Client: Event[EvTcpConnectFail] => State[Connect] -> State[Idle] 03/31/2025 02:57:31.506 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:31.506 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = cn-jenkins-deploy-platform-juju-os-494-1 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.0.0.17:2181, ] status = Up description = >>, << type = Collector name = Collector server_addrs = [ 10.0.0.17:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = Cassandra server_addrs = [ 10.0.0.17:9041, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 10.0.0.17:5673, ] status = Up description = >>, << type = Generic Connection name = Keystone server_addrs = [ http://252.17.31.96:5050/v3, ] status = Up description = >>, ] description = Collector:Collector[Connect to Idle on EvTcpConnectFail] connection down >>, ] >> 03/31/2025 02:57:31.506 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:31.506 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:31.507 7f0b58f140f0 [contrail-api] [ERROR]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = cn-jenkins-deploy-platform-juju-os-494-1:Config:contrail-api:0 client_info = << status = Idle successful_connections = 1 pid = 1 http_port = 8084 start_time = 1743387289504851 collector_name = collector_ip = 10.0.0.17:8086 collector_list = [ 10.0.0.17:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 4 >> 03/31/2025 02:57:31.507 7f0b58f140f0 [contrail-api] [INFO]: Discarding event[EvSandeshUVESend] in state[Idle] 03/31/2025 02:57:31.817 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:32.512 7f0b58f140f0 [contrail-api] [DEBUG]: send sync_uve in the [none:ConfigApiWorkerTrace] map 03/31/2025 02:57:32.564 7f0b58f140f0 [contrail-api] [DEBUG]: send sync_uve in the [ObjectConfigNode:NodeStatusUVE] map 03/31/2025 02:57:32.819 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:33.820 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:34.822 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:35.506 7f0b58f140f0 [contrail-api] [INFO]: Processing event[EvIdleHoldTimerExpired] in state[Idle] 03/31/2025 02:57:35.823 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:36.825 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:37.826 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process. 03/31/2025 02:57:38.828 7f0b595776d8 [contrail-api] [DEBUG]: __default__ [SYS_DEBUG]: VncApiError: KombuAmqpClient: socket.timeout occurred while draining events, no events to process.