2026-03-30 Mon 03:36:37:313.803 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: http-server-port 8091 2026-03-30 Mon 03:36:37:313.898 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Max-tasks 16 2026-03-30 Mon 03:36:37:313.906 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Max-slice 100 2026-03-30 Mon 03:36:37:313.913 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Collectors 10.0.0.249:8086 2026-03-30 Mon 03:36:37:313.918 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Collectors 10.0.0.241:8086 2026-03-30 Mon 03:36:37:313.923 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Collectors 10.0.0.141:8086 2026-03-30 Mon 03:36:37:313.941 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: ROLE : Generator 2026-03-30 Mon 03:36:37:313.948 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-30 Mon 03:36:37:313.953 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5563-2. 2026-03-30 Mon 03:36:37:313.958 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-30 Mon 03:36:37:313.964 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-30 Mon 03:36:37:313.969 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-30 Mon 03:36:37:313.975 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-30 Mon 03:36:37:314.539 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: HTTP Introspect Init 2026-03-30 Mon 03:36:37:314.702 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-03-30 Mon 03:36:37:314.762 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Sandesh Http Server Port: 8091 2026-03-30 Mon 03:36:37:314.798 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-30 Mon 03:36:37:314.807 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-30 Mon 03:36:37:314.881 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Idle 2026-03-30 Mon 03:36:37:315.033 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: No Client: 1774841797314961 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841797314891 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:36:37:315.167 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-30 Mon 03:36:37:315.181 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-30 Mon 03:36:37:315.193 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Cassandra Servers: 10.0.0.249:9042 10.0.0.241:9042 10.0.0.141:9042 2026-03-30 Mon 03:36:37:316.213 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Processing scm::EvStart in state Idle 2026-03-30 Mon 03:36:37:316.276 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Disconnect 2026-03-30 Mon 03:36:37:316.346 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-30 Mon 03:36:37:316.558 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Connect : Start Connect timer 10.0.0.241:8086 2026-03-30 Mon 03:36:37:316.639 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:37:316.660 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: SANDESH: Send FAILED: 1774841797316268 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:36:37:316.669 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:36:37:316.678 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:37:316.755 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: SANDESH: Send FAILED: 1774841797316336 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841797314891 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:36:37:316.771 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:36:37:316.781 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:37:318.736 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: SANDESH: Send FAILED: 1774841797316393 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841797314891 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:36:37:318.772 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:36:37:318.789 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:37:318.809 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: SANDESH: Send FAILED: 1774841797316414 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:36:37:318.816 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:36:37:318.824 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:37:318.895 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: SANDESH: Send FAILED: 1774841797316628 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841797314891 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:36:37:318.907 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:36:37:352.793 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-30 Mon 03:36:37:352.833 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:36:37:352.904 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:36:37:352.986 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422849361664, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:37:353.018 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422849361664, Pid 1]: SANDESH: Send FAILED: 1774841797352873 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:36:37:353.027 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422849361664, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:36:42:355.794 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-30 Mon 03:36:42:355.838 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:36:42:355.874 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:36:47:357.699 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-30 Mon 03:36:47:357.740 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:36:47:357.761 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:36:52:359.571 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-30 Mon 03:36:52:359.631 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:36:52:359.660 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:36:57:361.904 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-30 Mon 03:36:57:361.950 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:36:57:361.972 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:37:02:575.015 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-30 Mon 03:37:02:575.112 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:37:02:575.154 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:37:07:605.782 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-30 Mon 03:37:07:605.843 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:37:07:605.872 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:37:12:749.973 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-30 Mon 03:37:12:750.040 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:37:12:750.071 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:37:17:770.385 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-30 Mon 03:37:17:770.439 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:37:22:815.683 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-30 Mon 03:37:27:825.032 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:37:27:825.060 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:37:27:825.154 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:37:27:825.175 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x24791a0 2026-03-30 Mon 03:37:27:825.182 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.196 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.208 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x2479ae0 2026-03-30 Mon 03:37:27:825.214 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.222 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.232 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x247a420 2026-03-30 Mon 03:37:27:825.237 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.245 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.254 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x247ad60 2026-03-30 Mon 03:37:27:825.259 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.267 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.275 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x247b6a0 2026-03-30 Mon 03:37:27:825.280 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.297 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.306 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x24775f0 2026-03-30 Mon 03:37:27:825.311 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.319 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.328 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x2477ca0 2026-03-30 Mon 03:37:27:825.340 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.348 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.357 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x24711d0 2026-03-30 Mon 03:37:27:825.362 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.370 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.378 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x2478870 2026-03-30 Mon 03:37:27:825.383 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.391 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.399 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x2478340 2026-03-30 Mon 03:37:27:825.404 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.412 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.420 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x247bfe0 2026-03-30 Mon 03:37:27:825.426 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.433 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.442 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x247c920 2026-03-30 Mon 03:37:27:825.447 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.458 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.467 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x247d260 2026-03-30 Mon 03:37:27:825.475 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.482 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.491 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x247dba0 2026-03-30 Mon 03:37:27:825.496 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.504 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.512 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: 0 0x247e4e0 2026-03-30 Mon 03:37:27:825.517 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:27:825.525 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:27:825.742 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:825.760 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:825.804 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:825.820 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:825.830 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:825.844 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:825.853 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:825.865 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:825.874 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:825.887 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:825.896 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:825.932 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:825.952 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:825.969 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:825.123 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: SANDESH: Send FAILED: 1774841847824930 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:37:27:825.985 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:826.019 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:27:826.035 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:37:27:826.089 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:826.130 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:826.105 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:37:27:826.151 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:826.205 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:826.216 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:826.238 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:826.248 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:826.263 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:826.277 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:826.311 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:826.329 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:826.367 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:826.387 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:826.401 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:826.388 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:37:27:826.414 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:826.442 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:826.455 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:826.464 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:826.478 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:826.478 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Idle 2026-03-30 Mon 03:37:27:826.490 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:826.524 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:826.535 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:826.548 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:27:826.559 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:27:826.582 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:826.606 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:826.613 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:27:826.631 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:826.649 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:826.658 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: SANDESH: Send FAILED: 1774841847825795 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:37:27:826.664 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:826.678 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:27:826.695 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:27:826.710 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:27:826.741 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: SANDESH: Send FAILED: 1774841847825922 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.249:6379::Down.10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:37:27:826.753 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:27:826.784 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:27:826.813 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: SANDESH: Send FAILED: 1774841847826302 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:37:27:826.822 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:27:826.834 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:27:826.996 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: SANDESH: Send FAILED: 1774841847826307 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841797314891 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 65 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 66, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:37:27:827.014 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:37:27:827.029 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:27:827.057 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: SANDESH: Send FAILED: 1774841847826350 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:37:27:827.066 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:27:827.078 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:37:27:827.146 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:27:827.311 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: SANDESH: Send FAILED: 1774841847826598 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841797314891 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:37:27:827.334 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:37:27:827.346 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:27:827.377 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: SANDESH: Send FAILED: 1774841847826624 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:37:27:827.386 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422827845376, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:27:827.542 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422143801088, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:27:827.606 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422143801088, Pid 1]: SANDESH: Send FAILED: 1774841847827425 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774841847826475 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774841847827344 ] max_sm_queue_count = 7 ] 2026-03-30 Mon 03:37:27:827.615 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422143801088, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:37:27:827.626 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422143801088, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:27:827.740 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422143801088, Pid 1]: SANDESH: Send FAILED: 1774841847827502 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841797314891 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 14, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:37:27:827.759 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422143801088, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:37:28:826.672 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:37:28:826.975 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Connect : Start Connect timer 10.0.0.249:8086 2026-03-30 Mon 03:37:28:827.100 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:37:28:827.145 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: SANDESH: Send FAILED: 1774841848826809 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:37:28:827.154 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:28:827.188 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:37:28:827.312 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: SANDESH: Send FAILED: 1774841848827085 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841797314891 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:37:28:827.332 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:37:28:829.260 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:52558::10.0.0.249:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:37:28:829.299 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: 10.0.0.241:52558::10.0.0.249:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:37:28:829.308 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: 10.0.0.241:52558::10.0.0.249:8086(-1):EvTcpConnected 2026-03-30 Mon 03:37:28:829.417 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:37:28:829.454 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:37:28:829.499 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: ClientInit 2026-03-30 Mon 03:37:28:829.523 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-2.:contrail-query-engine:0:Database count 1 2026-03-30 Mon 03:37:28:829.614 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:37:28:829.624 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:37:28:829.662 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: SANDESH: Send FAILED: 1774841848829489 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:37:28:829.671 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:37:28:829.682 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:37:28:829.692 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:37:28:829.705 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:37:28:829.711 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:37:28:829.824 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: SANDESH: Send FAILED: 1774841848829602 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774841797314891 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:37:28:829.841 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422857893632, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:37:28:836.902 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422836238080, Pid 1]: OnMessage control in state: ClientInit 2026-03-30 Mon 03:37:28:836.983 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422836238080, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-30 Mon 03:37:28:836.994 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422836238080, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-30 Mon 03:37:28:837.019 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422836238080, Pid 1]: Received Ctrl Message with size 0 2026-03-30 Mon 03:37:28:837.193 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422836238080, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774841797314891 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774841847826475 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774841847827344 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 7 proxy seq 10 2026-03-30 Mon 03:37:28:837.227 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422836238080, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:37:28:837.254 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422836238080, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2026-03-30 Mon 03:37:28:837.266 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422836238080, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:37:28:837.273 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422836238080, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:37:28:837.281 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422836238080, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:37:28:837.315 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422836238080, Pid 1]: Established 2026-03-30 Mon 03:37:35:152.228 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422853560064, Pid 1]: OnMessage control in state: Established 2026-03-30 Mon 03:37:35:152.325 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-03-30 Mon 03:37:35:152.358 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Established : EvSandeshMessageRecv 2026-03-30 Mon 03:37:35:152.410 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: Received Ctrl Message with size 3 2026-03-30 Mon 03:37:35:152.424 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2026-03-30 Mon 03:37:35:152.435 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2026-03-30 Mon 03:37:35:152.447 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2026-03-30 Mon 03:37:35:152.455 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422883084032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:37:57:828.343 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422139602688, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-30 Mon 03:38:57:828.873 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422143801088, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-30 Mon 03:39:57:829.541 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422874687232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-30 Mon 03:40:21:673.239 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:8091::10.0.0.241:47102 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-30 Mon 03:40:54:254.680 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:8091::10.0.0.241:56090 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-30 Mon 03:40:57:830.045 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422849361664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-30 Mon 03:41:57:830.653 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422870488832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-30 Mon 03:42:57:831.785 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422849361664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-30 Mon 03:43:19:097.290 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Disconnect status: -1 0x247b6a0 2026-03-30 Mon 03:43:19:097.353 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:097.409 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Disconnect status: -1 0x247ad60 2026-03-30 Mon 03:43:19:097.422 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:097.554 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Disconnect status: -1 0x247a420 2026-03-30 Mon 03:43:19:097.572 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:097.599 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Disconnect status: -1 0x2479ae0 2026-03-30 Mon 03:43:19:097.610 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:097.652 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-30 Mon 03:43:19:097.660 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Disconnect status: -1 0x24791a0 2026-03-30 Mon 03:43:19:097.672 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:103.192 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: -1 0x247b6a0 2026-03-30 Mon 03:43:19:103.258 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: -1 0x247ad60 2026-03-30 Mon 03:43:19:103.308 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: -1 0x247a420 2026-03-30 Mon 03:43:19:103.343 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: -1 0x2479ae0 2026-03-30 Mon 03:43:19:103.364 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140423030954112, Pid 1]: RAC_Connect status: -1 0x24791a0 2026-03-30 Mon 03:43:19:231.455 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422853560064, Pid 1]: Processing scm::EvStop in state Established 2026-03-30 Mon 03:43:19:231.521 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422853560064, Pid 1]: Idle 2026-03-30 Mon 03:43:19:231.628 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422853560064, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:43:19:231.640 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422853560064, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:43:19:231.811 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422853560064, Pid 1]: SANDESH: Send FAILED: 1774842199231615 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774841797314891 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 308930 calls = 34 average_bytes = 9086 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 93 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 24 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 35, ] ] ] 2026-03-30 Mon 03:43:19:231.833 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140422853560064, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:43:32:852.650 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: http-server-port 8091 2026-03-30 Mon 03:43:32:852.733 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Max-tasks 16 2026-03-30 Mon 03:43:32:852.742 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Max-slice 100 2026-03-30 Mon 03:43:32:852.754 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Collectors 10.0.0.249:8086 2026-03-30 Mon 03:43:32:852.760 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Collectors 10.0.0.241:8086 2026-03-30 Mon 03:43:32:852.765 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Collectors 10.0.0.141:8086 2026-03-30 Mon 03:43:32:852.784 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: ROLE : Generator 2026-03-30 Mon 03:43:32:852.790 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-30 Mon 03:43:32:852.795 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5563-2. 2026-03-30 Mon 03:43:32:852.800 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-30 Mon 03:43:32:852.805 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-30 Mon 03:43:32:852.810 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-30 Mon 03:43:32:852.816 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-30 Mon 03:43:32:853.366 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: HTTP Introspect Init 2026-03-30 Mon 03:43:32:853.463 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-03-30 Mon 03:43:32:853.516 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Sandesh Http Server Port: 8091 2026-03-30 Mon 03:43:32:853.544 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-30 Mon 03:43:32:853.553 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-30 Mon 03:43:32:853.625 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Idle 2026-03-30 Mon 03:43:32:853.772 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: No Client: 1774842212853701 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:43:32:853.900 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-30 Mon 03:43:32:853.914 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-30 Mon 03:43:32:853.926 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Cassandra Servers: 10.0.0.249:9042 10.0.0.241:9042 10.0.0.141:9042 2026-03-30 Mon 03:43:32:854.004 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Processing scm::EvStart in state Idle 2026-03-30 Mon 03:43:32:854.073 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Disconnect 2026-03-30 Mon 03:43:32:854.144 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-30 Mon 03:43:32:854.490 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Connect : Start Connect timer 10.0.0.141:8086 2026-03-30 Mon 03:43:32:854.562 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:43:32:854.583 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: SANDESH: Send FAILED: 1774842212854065 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:43:32:854.591 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:43:32:854.600 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:43:32:854.677 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: SANDESH: Send FAILED: 1774842212854134 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:43:32:854.694 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:43:32:854.703 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:43:32:854.772 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: SANDESH: Send FAILED: 1774842212854190 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:43:32:854.800 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:43:32:854.810 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:43:32:854.825 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: SANDESH: Send FAILED: 1774842212854210 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:43:32:854.832 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:43:32:854.839 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:43:32:854.902 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: SANDESH: Send FAILED: 1774842212854550 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:43:32:854.914 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:43:32:914.721 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-30 Mon 03:44:07:949.638 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:44:07:949.681 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:44:07:949.709 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:44:07:949.731 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a515f0 2026-03-30 Mon 03:44:07:949.738 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:07:949.753 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:07:949.766 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a51ca0 2026-03-30 Mon 03:44:07:949.771 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:07:949.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:07:949.789 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a531a0 2026-03-30 Mon 03:44:07:949.795 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:07:949.803 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:07:949.812 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a52340 2026-03-30 Mon 03:44:07:949.818 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:07:949.825 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:07:949.841 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a4b1d0 2026-03-30 Mon 03:44:07:949.846 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:07:949.855 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:07:949.862 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a52870 2026-03-30 Mon 03:44:07:949.868 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:07:949.875 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:07:949.884 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a53ae0 2026-03-30 Mon 03:44:07:949.889 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:07:949.897 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:07:949.905 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a54420 2026-03-30 Mon 03:44:07:949.910 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:07:949.918 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:07:949.927 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a54d60 2026-03-30 Mon 03:44:07:949.932 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:07:949.939 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:07:949.948 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a556a0 2026-03-30 Mon 03:44:07:949.953 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:07:949.960 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:07:949.974 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:44:07:950.006 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:44:07:949.709 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842247949538 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:44:07:950.030 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:44:07:950.047 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:44:07:950.062 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:44:07:950.081 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:44:07:950.089 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:44:07:950.090 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:44:07:950.245 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:44:07:950.285 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Idle 2026-03-30 Mon 03:44:07:950.352 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:44:07:950.401 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:07:950.438 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:07:950.461 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842247950185 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 12, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:44:07:950.480 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:44:07:950.486 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:07:950.490 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:44:07:950.525 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:07:950.525 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842247950211 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:44:07:950.546 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:44:07:950.558 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:44:07:950.581 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:07:950.592 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:07:950.621 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:07:950.630 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:07:950.642 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:07:950.596 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:44:07:950.657 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:07:950.721 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:07:950.757 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:07:950.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:07:950.786 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842247950341 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 13, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:44:07:950.796 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:07:950.806 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:44:07:950.828 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:07:950.830 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:44:07:950.854 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:07:950.863 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842247950710 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:44:07:950.876 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:07:950.887 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:44:07:950.906 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:07:950.910 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:44:07:950.921 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:07:950.943 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842247950749 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.249:6379::Down.10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:44:07:950.945 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:07:950.955 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:44:07:950.969 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:07:950.994 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:07:951.004 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:07:951.021 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:07:951.032 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:07:951.061 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:44:07:951.064 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:07:951.116 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842247950995 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842247950284 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842247950908 ] max_sm_queue_count = 5 ] 2026-03-30 Mon 03:44:07:951.125 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:44:07:951.134 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:44:07:951.134 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:07:951.166 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:07:951.187 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:07:951.207 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:07:951.273 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842247951048 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 36 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 38, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 11, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:44:07:951.289 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:44:07:951.301 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:44:07:951.322 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842247951126 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:44:07:951.329 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:44:08:950.424 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:44:08:950.651 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Connect : Start Connect timer 10.0.0.241:8086 2026-03-30 Mon 03:44:08:950.746 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:44:08:950.775 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: SANDESH: Send FAILED: 1774842248950495 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:44:08:950.784 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:44:08:950.793 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:44:08:950.903 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: SANDESH: Send FAILED: 1774842248950734 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 14, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:44:08:950.923 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:44:08:954.709 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:60466::10.0.0.241:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:44:08:954.738 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:60466::10.0.0.241:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:44:08:954.746 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:60466::10.0.0.241:8086(-1):EvTcpConnected 2026-03-30 Mon 03:44:08:954.806 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:44:08:954.831 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:44:08:954.872 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: ClientInit 2026-03-30 Mon 03:44:08:954.893 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-2.:contrail-query-engine:0:Database count 1 2026-03-30 Mon 03:44:08:954.976 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:44:08:954.985 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:44:08:955.008 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842248954864 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:44:08:955.015 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:44:08:955.024 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:44:08:955.031 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:44:08:955.043 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:44:08:955.050 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:44:08:955.151 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842248954967 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 44, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:44:08:955.170 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:44:08:980.934 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: OnMessage control in state: ClientInit 2026-03-30 Mon 03:44:08:980.997 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-30 Mon 03:44:08:981.010 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-30 Mon 03:44:08:981.034 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: Received Ctrl Message with size 0 2026-03-30 Mon 03:44:08:981.187 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842247950284 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842247950908 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 44, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2026-03-30 Mon 03:44:08:981.214 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:44:08:981.248 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 9 2026-03-30 Mon 03:44:08:981.261 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:44:08:981.269 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:44:08:981.276 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:44:08:981.310 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: Established 2026-03-30 Mon 03:44:12:950.287 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:44:12:950.406 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:44:12:950.443 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:44:12:950.473 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:44:12:950.503 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:44:12:950.662 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a56920 2026-03-30 Mon 03:44:12:950.673 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:12:950.697 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:12:950.721 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a55fe0 2026-03-30 Mon 03:44:12:950.731 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:12:950.739 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:12:950.748 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a57260 2026-03-30 Mon 03:44:12:950.754 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:12:950.761 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:12:950.772 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a57ba0 2026-03-30 Mon 03:44:12:950.778 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:12:950.785 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:12:950.819 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a584e0 2026-03-30 Mon 03:44:12:950.825 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:44:12:950.837 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:44:12:951.124 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:12:951.145 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:12:951.171 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:12:951.195 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:12:951.209 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:12:951.231 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:12:951.241 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:12:951.254 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:12:951.262 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:12:951.275 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:44:12:951.284 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:44:12:951.332 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:12:951.350 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:12:951.365 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:12:951.385 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:44:37:951.606 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-30 Mon 03:45:37:952.623 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-30 Mon 03:46:32:572.275 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a556a0 2026-03-30 Mon 03:46:32:572.319 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:572.365 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a54d60 2026-03-30 Mon 03:46:32:572.377 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:572.410 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a54420 2026-03-30 Mon 03:46:32:572.421 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:572.446 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a53ae0 2026-03-30 Mon 03:46:32:572.456 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:572.484 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-30 Mon 03:46:32:572.492 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a531a0 2026-03-30 Mon 03:46:32:572.501 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:572.554 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:60466::10.0.0.241:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-30 Mon 03:46:32:572.566 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:60466::10.0.0.241:8086(-1):EvTcpClose 2026-03-30 Mon 03:46:32:573.195 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvTcpClose in state Established 2026-03-30 Mon 03:46:32:573.212 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Established : EvTcpClose 2026-03-30 Mon 03:46:32:573.460 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Connect : Start Connect timer 10.0.0.249:8086 2026-03-30 Mon 03:46:32:573.548 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:32:573.707 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842392573296 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = cn-jenkins-deploy-platform-ansible-os-5563-2. collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 142424 calls = 17 average_bytes = 8377 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 70, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 18, ] ] ] 2026-03-30 Mon 03:46:32:573.734 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:573.746 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:32:573.769 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842392573325 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:46:32:573.776 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:32:573.785 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2026-03-30 Mon 03:46:32:573.794 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:32:573.918 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842392573535 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 142424 calls = 17 average_bytes = 8377 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 70, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 18, ] ] ] 2026-03-30 Mon 03:46:32:573.936 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:574.885 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:32:574.927 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842392574521 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.241:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:46:32:574.936 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:32:584.331 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:40868::10.0.0.249:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:46:32:584.365 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:40868::10.0.0.249:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:46:32:584.373 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:40868::10.0.0.249:8086(-1):EvTcpConnected 2026-03-30 Mon 03:46:32:585.023 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:46:32:585.046 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:46:32:585.085 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: ClientInit 2026-03-30 Mon 03:46:32:585.101 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-2.:contrail-query-engine:0:Database count 2 2026-03-30 Mon 03:46:32:585.213 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:32:585.230 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:46:32:585.254 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842392585077 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.241:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:46:32:585.262 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:46:32:585.271 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:32:585.278 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:46:32:585.289 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:32:585.295 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:585.432 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842392585200 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 142424 calls = 17 average_bytes = 8377 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 71, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 22, _iter59->first = sent _iter59->second = 18, ] ] ] 2026-03-30 Mon 03:46:32:585.448 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:586.271 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:46:32:586.314 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:46:32:586.330 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:46:32:586.344 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:46:32:586.358 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:46:32:620.081 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: OnMessage control in state: ClientInit 2026-03-30 Mon 03:46:32:620.192 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-30 Mon 03:46:32:620.228 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-30 Mon 03:46:32:620.260 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: Received Ctrl Message with size 0 2026-03-30 Mon 03:46:32:620.538 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 142424 calls = 17 average_bytes = 8377 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 33 dequeues = 33 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 40 dequeues = 40 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Established state_since = 1774842248981308 last_state = Established last_event = scm::EvSandeshSend last_event_at = 1774842337952676 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 15 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 117070 calls = 14 average_bytes = 8362 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 71, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 22, _iter59->first = sent _iter59->second = 18, ] ] max_sm_queue_count = 5 proxy seq 22 2026-03-30 Mon 03:46:32:620.590 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:46:32:620.616 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.241:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 15 2026-03-30 Mon 03:46:32:620.629 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:46:32:620.662 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.121306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 3 2026-03-30 Mon 03:46:32:620.672 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:46:32:620.683 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:46:32:620.715 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: Established 2026-03-30 Mon 03:46:32:694.715 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a52870 2026-03-30 Mon 03:46:32:694.775 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:694.835 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a52340 2026-03-30 Mon 03:46:32:694.852 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:695.025 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a4b1d0 2026-03-30 Mon 03:46:32:695.043 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:695.077 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-30 Mon 03:46:32:695.086 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a515f0 2026-03-30 Mon 03:46:32:695.099 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:695.227 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a51ca0 2026-03-30 Mon 03:46:32:695.244 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:696.319 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:46:32:696.357 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:46:32:696.407 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:46:32:696.426 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:46:32:696.499 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:46:32:741.224 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a584e0 2026-03-30 Mon 03:46:32:741.271 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:741.619 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a57260 2026-03-30 Mon 03:46:32:741.638 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:741.667 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a57ba0 2026-03-30 Mon 03:46:32:741.677 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:741.696 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a56920 2026-03-30 Mon 03:46:32:741.706 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:741.730 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-30 Mon 03:46:32:741.737 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Disconnect status: -1 0x1a55fe0 2026-03-30 Mon 03:46:32:741.745 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:741.969 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:46:32:742.091 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:46:32:742.111 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:46:32:742.124 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:46:32:742.137 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:46:32:834.888 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.241:40868::10.0.0.249:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:46:32:834.939 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:40868::10.0.0.249:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-30 Mon 03:46:32:834.948 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:40868::10.0.0.249:8086(-1):EvTcpClose 2026-03-30 Mon 03:46:32:835.031 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Processing scm::EvTcpClose in state Established 2026-03-30 Mon 03:46:32:835.067 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Established : EvTcpClose 2026-03-30 Mon 03:46:32:835.360 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Connect : Start Connect timer 10.0.0.141:8086 2026-03-30 Mon 03:46:32:835.471 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:32:835.650 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: SANDESH: Send FAILED: 1774842392835154 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = cn-jenkins-deploy-platform-ansible-os-5563-1. collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 198676 calls = 23 average_bytes = 8638 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-03-30 Mon 03:46:32:835.676 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:835.688 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:32:835.711 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: SANDESH: Send FAILED: 1774842392835214 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:32:835.718 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:32:835.728 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2026-03-30 Mon 03:46:32:835.791 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:32:835.948 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: SANDESH: Send FAILED: 1774842392835446 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 198676 calls = 23 average_bytes = 8638 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-03-30 Mon 03:46:32:835.966 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:836.121 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:46:32:836.133 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:46:32:836.340 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:46:32:836.357 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:46:32:836.493 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:46:32:836.522 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Idle 2026-03-30 Mon 03:46:32:836.647 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:32:836.798 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: SANDESH: Send FAILED: 1774842392836443 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 198676 calls = 23 average_bytes = 8638 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-03-30 Mon 03:46:32:836.820 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:836.839 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:32:836.861 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: SANDESH: Send FAILED: 1774842392836472 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:32:836.868 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:32:836.877 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:46:32:836.886 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:32:837.017 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: SANDESH: Send FAILED: 1774842392836633 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 198676 calls = 23 average_bytes = 8638 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-03-30 Mon 03:46:32:837.033 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:33:836.654 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:46:33:837.000 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Connect : Start Connect timer 10.0.0.241:8086 2026-03-30 Mon 03:46:33:837.152 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:33:837.208 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: SANDESH: Send FAILED: 1774842393836747 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:33:837.238 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:33:837.254 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:33:837.596 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: SANDESH: Send FAILED: 1774842393837126 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 198676 calls = 23 average_bytes = 8638 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-03-30 Mon 03:46:33:837.622 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:33:839.307 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:48264::10.0.0.241:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:46:33:839.349 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:48264::10.0.0.241:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:46:33:839.359 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:48264::10.0.0.241:8086(-1):EvTcpConnected 2026-03-30 Mon 03:46:33:839.740 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:46:33:839.769 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:46:33:839.814 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: ClientInit 2026-03-30 Mon 03:46:33:839.856 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-2.:contrail-query-engine:0:Database count 3 2026-03-30 Mon 03:46:33:839.967 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:33:839.980 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:46:33:840.008 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: SANDESH: Send FAILED: 1774842393839806 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:33:840.020 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:46:33:840.031 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:33:840.038 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:46:33:840.049 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:33:840.056 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:46:33:840.242 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: SANDESH: Send FAILED: 1774842393839948 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 198676 calls = 23 average_bytes = 8638 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-03-30 Mon 03:46:33:840.262 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719332718336, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:46:33:841.238 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:48264::10.0.0.241:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-30 Mon 03:46:33:841.260 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:48264::10.0.0.241:8086(-1):EvTcpClose 2026-03-30 Mon 03:46:33:841.550 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-30 Mon 03:46:33:841.567 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: ClientInit : EvTcpClose 2026-03-30 Mon 03:46:33:841.694 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Return to idle with 1000 2026-03-30 Mon 03:46:33:841.704 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Idle 2026-03-30 Mon 03:46:33:841.763 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:33:841.899 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: SANDESH: Send FAILED: 1774842393841646 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200249 calls = 24 average_bytes = 8343 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 34, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-30 Mon 03:46:33:841.922 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:33:841.932 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:33:841.954 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: SANDESH: Send FAILED: 1774842393841670 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:33:841.961 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:33:841.970 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:46:33:841.979 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:33:842.112 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: SANDESH: Send FAILED: 1774842393841754 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200249 calls = 24 average_bytes = 8343 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 34, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-30 Mon 03:46:33:842.127 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:34:842.220 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:46:34:842.451 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Connect : Start Connect timer 10.0.0.249:8086 2026-03-30 Mon 03:46:34:842.548 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:34:842.576 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842394842296 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:34:842.585 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:34:842.594 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:34:842.735 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842394842532 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200249 calls = 24 average_bytes = 8343 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-30 Mon 03:46:34:842.759 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:34:844.732 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:40946::10.0.0.249:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:46:34:844.753 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:40946::10.0.0.249:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:46:34:844.761 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:40946::10.0.0.249:8086(-1):EvTcpConnected 2026-03-30 Mon 03:46:34:844.957 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:46:34:844.968 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:46:34:844.998 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: ClientInit 2026-03-30 Mon 03:46:34:845.009 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-2.:contrail-query-engine:0:Database count 4 2026-03-30 Mon 03:46:34:845.086 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:34:845.095 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:46:34:845.116 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: SANDESH: Send FAILED: 1774842394844992 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:34:845.124 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:46:34:845.133 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:34:845.146 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:46:34:845.157 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:34:845.163 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:46:34:845.300 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: SANDESH: Send FAILED: 1774842394845075 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200249 calls = 24 average_bytes = 8343 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 39, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-30 Mon 03:46:34:845.315 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:46:37:586.374 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:46:37:586.536 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:46:37:586.575 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:46:37:586.621 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:46:37:586.648 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:46:37:586.674 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:46:37:586.684 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:46:37:586.715 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:46:37:586.748 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:46:37:586.763 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:46:37:696.426 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:46:37:696.605 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:46:37:696.660 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:46:37:696.691 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:46:37:696.726 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:46:37:696.891 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:46:37:696.995 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:46:37:697.015 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:46:37:697.029 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:46:37:697.044 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:46:37:742.029 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:46:37:742.161 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:46:37:742.213 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:46:37:742.242 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:46:37:742.269 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:46:37:742.605 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:46:37:742.656 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:46:37:742.672 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:46:37:742.685 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:46:37:742.698 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:46:37:953.247 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-30 Mon 03:46:37:953.333 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:37:953.374 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:46:37:953.475 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: SANDESH: Send FAILED: 1774842397953124 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 70 dequeues = 70 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 91 dequeues = 91 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1774842394844997 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1774842394845155 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 6 ] 2026-03-30 Mon 03:46:37:953.490 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:46:37:953.501 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:37:953.509 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:46:37:953.636 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: SANDESH: Send FAILED: 1774842397953225 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 201822 calls = 25 average_bytes = 8072 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 41, _iter59->first = sent _iter59->second = 29, ] ] ] 2026-03-30 Mon 03:46:37:953.665 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:46:38:845.152 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: 10.0.0.249:8086 EvConnectTimerExpired in state ClientInit 2026-03-30 Mon 03:46:38:845.225 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2026-03-30 Mon 03:46:38:845.239 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: ClientInit : EvConnectTimerExpired 2026-03-30 Mon 03:46:38:845.419 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Return to idle with 1000 2026-03-30 Mon 03:46:38:845.431 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Idle 2026-03-30 Mon 03:46:38:845.501 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:38:845.663 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: SANDESH: Send FAILED: 1774842398845321 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 201822 calls = 25 average_bytes = 8072 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 29, ] ] ] 2026-03-30 Mon 03:46:38:845.707 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:38:845.720 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:38:845.744 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: SANDESH: Send FAILED: 1774842398845354 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:38:845.751 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:38:845.760 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:46:38:845.789 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:38:845.922 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: SANDESH: Send FAILED: 1774842398845487 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 201822 calls = 25 average_bytes = 8072 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 29, ] ] ] 2026-03-30 Mon 03:46:38:845.937 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:39:846.144 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719349511936, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:46:39:846.423 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719349511936, Pid 1]: Connect : Start Connect timer 10.0.0.141:8086 2026-03-30 Mon 03:46:39:846.588 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719349511936, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:39:846.630 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719349511936, Pid 1]: SANDESH: Send FAILED: 1774842399846254 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:39:846.643 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719349511936, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:39:846.656 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719349511936, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:39:846.906 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719349511936, Pid 1]: SANDESH: Send FAILED: 1774842399846568 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 201822 calls = 25 average_bytes = 8072 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 46, _iter59->first = sent _iter59->second = 29, ] ] ] 2026-03-30 Mon 03:46:39:846.940 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719349511936, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:39:849.945 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:41798::10.0.0.141:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:46:39:849.987 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:41798::10.0.0.141:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:46:39:849.999 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:41798::10.0.0.141:8086(-1):EvTcpConnected 2026-03-30 Mon 03:46:39:850.312 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:46:39:850.333 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:46:39:850.382 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: ClientInit 2026-03-30 Mon 03:46:39:850.405 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-2.:contrail-query-engine:0:Database count 5 2026-03-30 Mon 03:46:39:850.529 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:39:850.545 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:46:39:850.578 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842399850370 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:39:850.601 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:46:39:850.615 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:39:850.626 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:46:39:850.641 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:39:850.652 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:46:39:850.887 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842399850513 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 201822 calls = 25 average_bytes = 8072 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 29, ] ] ] 2026-03-30 Mon 03:46:39:850.916 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:46:39:852.863 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:41798::10.0.0.141:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-30 Mon 03:46:39:852.895 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:41798::10.0.0.141:8086(-1):EvTcpClose 2026-03-30 Mon 03:46:39:853.726 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-30 Mon 03:46:39:853.744 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: ClientInit : EvTcpClose 2026-03-30 Mon 03:46:39:853.920 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Return to idle with 1000 2026-03-30 Mon 03:46:39:853.935 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Idle 2026-03-30 Mon 03:46:39:854.028 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:39:854.283 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: SANDESH: Send FAILED: 1774842399853846 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203395 calls = 26 average_bytes = 7822 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 30, ] ] ] 2026-03-30 Mon 03:46:39:854.310 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:39:854.325 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:39:854.358 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: SANDESH: Send FAILED: 1774842399853882 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:39:854.371 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:39:854.384 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:46:39:854.397 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:39:854.606 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: SANDESH: Send FAILED: 1774842399854014 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203395 calls = 26 average_bytes = 7822 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 30, ] ] ] 2026-03-30 Mon 03:46:39:854.630 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:40:854.527 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:46:40:854.770 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Connect : Start Connect timer 10.0.0.241:8086 2026-03-30 Mon 03:46:40:854.865 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:40:854.906 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: SANDESH: Send FAILED: 1774842400854602 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:40:854.915 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:40:854.924 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:40:855.072 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: SANDESH: Send FAILED: 1774842400854854 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203395 calls = 26 average_bytes = 7822 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 30, ] ] ] 2026-03-30 Mon 03:46:40:855.088 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:40:857.247 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:54120::10.0.0.241:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:46:40:857.272 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:54120::10.0.0.241:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:46:40:857.280 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:54120::10.0.0.241:8086(-1):EvTcpConnected 2026-03-30 Mon 03:46:40:857.946 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:46:40:857.959 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:46:40:857.995 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: ClientInit 2026-03-30 Mon 03:46:40:858.011 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-2.:contrail-query-engine:0:Database count 6 2026-03-30 Mon 03:46:40:858.092 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:40:858.101 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:46:40:858.124 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: SANDESH: Send FAILED: 1774842400857988 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:40:858.131 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:46:40:858.140 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:40:858.147 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:46:40:858.157 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:40:858.163 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:46:40:858.308 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: SANDESH: Send FAILED: 1774842400858083 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203395 calls = 26 average_bytes = 7822 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 55, _iter59->first = sent _iter59->second = 30, ] ] ] 2026-03-30 Mon 03:46:40:858.330 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140718832088832, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:46:42:586.689 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:46:42:586.852 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:46:42:586.885 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:46:42:586.926 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:46:42:586.962 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:46:42:587.000 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:46:42:587.049 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:46:42:587.071 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:46:42:587.086 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:46:42:587.099 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:46:42:696.991 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:46:42:697.151 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:46:42:697.262 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:46:42:697.307 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:46:42:697.337 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:46:42:697.457 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:46:42:697.523 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:46:42:697.543 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:46:42:697.558 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:46:42:723.919 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:46:42:742.713 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:46:42:742.861 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:46:42:742.893 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:46:42:742.927 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:46:42:742.969 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:46:42:743.321 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:46:42:743.412 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:46:42:743.429 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:46:42:743.444 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:46:42:743.460 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:46:44:466.721 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.241:54120::10.0.0.241:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2026-03-30 Mon 03:46:44:466.764 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.241:54120::10.0.0.241:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:46:44:466.785 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:54120::10.0.0.241:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-30 Mon 03:46:44:466.793 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:54120::10.0.0.241:8086(-1):EvTcpClose 2026-03-30 Mon 03:46:44:467.457 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-30 Mon 03:46:44:467.506 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: ClientInit : EvTcpClose 2026-03-30 Mon 03:46:44:467.677 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Return to idle with 1000 2026-03-30 Mon 03:46:44:467.698 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Idle 2026-03-30 Mon 03:46:44:467.771 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:44:467.929 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: SANDESH: Send FAILED: 1774842404467607 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:46:44:467.946 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:44:468.007 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:44:468.032 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: SANDESH: Send FAILED: 1774842404467654 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:44:468.043 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:44:468.054 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:46:44:468.086 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:44:468.335 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: SANDESH: Send FAILED: 1774842404467759 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:46:44:468.364 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:45:468.719 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:46:45:468.975 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Connect : Start Connect timer 10.0.0.249:8086 2026-03-30 Mon 03:46:45:469.079 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:45:469.108 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: SANDESH: Send FAILED: 1774842405468806 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:45:469.131 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:45:469.142 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:45:469.324 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:46:45:469.359 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:46:45:469.338 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: SANDESH: Send FAILED: 1774842405469067 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:46:45:469.391 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:45:469.403 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:46:45:469.414 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:46:45:469.509 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:46:45:469.543 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Idle 2026-03-30 Mon 03:46:45:469.599 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:45:469.737 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: SANDESH: Send FAILED: 1774842405469468 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 62, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:46:45:469.753 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:45:469.763 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:45:469.784 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: SANDESH: Send FAILED: 1774842405469488 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:45:469.793 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:45:469.802 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:46:45:469.820 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:45:469.945 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: SANDESH: Send FAILED: 1774842405469591 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 62, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:46:45:469.959 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719353710336, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:47:586.936 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:46:47:587.121 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:46:47:587.150 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:46:47:587.211 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:46:47:587.243 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:46:47:587.281 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:46:47:587.299 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:46:47:587.312 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:46:47:587.322 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:46:47:587.353 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:46:47:697.555 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:46:47:697.692 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:46:47:697.725 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:46:47:697.754 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:46:47:697.961 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:46:47:698.019 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:46:47:698.034 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:46:47:698.048 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:46:47:724.056 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:46:47:724.461 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:46:47:743.444 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:46:47:743.572 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:46:47:743.604 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:46:47:743.656 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:46:47:743.682 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:46:47:743.946 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:46:47:744.003 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:46:47:744.018 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:46:47:744.042 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:46:47:744.055 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:46:52:587.190 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:46:52:587.350 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:46:52:587.384 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:46:52:587.418 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:46:52:587.446 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:46:52:587.481 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:46:52:587.496 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:46:52:587.508 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:46:52:587.518 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:46:52:587.547 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:46:52:698.069 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:46:52:698.237 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:46:52:698.280 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:46:52:698.499 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:46:52:698.620 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:46:52:698.654 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:46:52:698.787 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:46:52:698.843 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:46:52:724.558 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:46:52:725.040 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:46:52:744.067 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:46:52:744.224 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:46:52:744.263 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:46:52:744.290 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:46:52:744.328 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:46:52:744.620 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:46:52:744.686 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:46:52:744.702 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:46:52:744.715 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:46:52:744.729 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:46:57:587.456 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:46:57:587.735 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:46:57:587.770 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:46:57:587.810 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:46:57:587.842 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:46:57:587.883 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:46:57:587.919 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:46:57:587.945 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:46:57:587.963 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:46:57:587.975 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:46:57:698.707 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:46:57:698.917 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:46:57:698.980 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:46:57:699.012 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:46:57:699.461 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:46:57:699.531 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:46:57:699.560 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:46:57:699.575 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:46:57:725.187 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:46:57:725.816 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:46:57:744.744 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:46:57:744.996 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:46:57:745.052 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:46:57:745.098 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:46:57:745.136 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:46:57:745.400 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:46:57:745.545 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:46:57:745.599 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:46:57:745.615 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:46:57:745.704 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:02:587.810 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:02:587.985 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:02:588.015 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:02:588.051 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:02:588.087 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:02:588.103 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:02:588.113 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:02:588.139 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:02:588.179 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:02:588.194 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:02:699.556 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:02:699.712 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:02:699.748 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:02:699.775 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:02:700.123 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:02:700.179 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:02:700.198 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:02:700.252 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:02:725.930 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:02:726.634 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:02:745.484 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:02:745.643 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:02:745.675 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:02:745.699 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:02:745.744 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:02:746.108 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:02:746.187 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:02:746.212 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:47:02:746.230 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:02:746.248 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:07:588.064 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:07:588.240 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:07:588.273 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:07:588.309 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:07:588.338 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:07:588.365 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:07:588.397 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:07:588.415 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:07:588.428 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:07:588.440 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:07:700.210 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:07:700.370 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:07:700.403 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:07:700.433 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:07:700.771 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:07:700.828 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:07:700.844 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:07:700.857 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:07:726.718 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:07:727.284 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:07:746.212 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:07:746.372 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:07:746.411 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:07:746.437 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:07:746.463 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:07:746.729 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:07:746.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:07:746.795 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:47:07:746.810 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:07:746.949 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:07:953.565 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:07:953.673 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: SANDESH: Send FAILED: 1774842427953430 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 96 dequeues = 96 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 130 dequeues = 130 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842405469541 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842405469818 ] max_sm_queue_count = 6 ] 2026-03-30 Mon 03:47:07:953.701 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:07:953.712 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:07:953.852 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: SANDESH: Send FAILED: 1774842427953525 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 65, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:47:07:953.866 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:12:588.311 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:12:588.489 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:12:588.519 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:12:588.706 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:12:588.739 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:12:588.766 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:12:588.801 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:12:588.819 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:12:588.831 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:12:588.843 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:12:700.866 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:12:701.017 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:12:701.060 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:12:701.090 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:12:701.309 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:12:701.338 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:12:701.353 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:12:701.366 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:12:727.380 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:12:727.872 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:12:746.834 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:12:746.995 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:12:747.065 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:12:747.109 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:12:747.165 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:12:747.364 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:12:747.438 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:12:747.462 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:47:12:747.496 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:12:747.517 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:12:958.848 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:12:958.945 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: SANDESH: Send FAILED: 1774842432958814 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.121306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-30 Mon 03:47:12:958.957 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719362107136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-03-30 Mon 03:47:15:470.331 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:47:15:470.529 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Connect : Start Connect timer 10.0.0.141:8086 2026-03-30 Mon 03:47:15:471.179 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:47:15:471.205 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:47:15:471.239 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:47:15:471.274 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842435470404 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:47:15:471.284 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:47:15:471.294 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:47:15:471.302 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:47:15:471.395 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:47:15:471.428 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Idle 2026-03-30 Mon 03:47:15:471.515 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:15:471.687 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842435471224 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:47:15:471.705 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:15:471.716 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:15:471.855 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842435471352 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 69, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:47:15:471.870 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:15:471.880 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:15:471.903 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842435471374 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:47:15:471.910 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:47:15:471.920 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:47:15:471.941 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:15:472.078 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842435471493 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 93, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 69, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:47:15:472.098 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:17:588.605 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:17:588.831 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:17:588.871 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:17:588.907 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:17:588.938 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:17:588.968 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:17:589.003 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:17:589.018 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:17:589.031 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:17:589.044 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:17:701.385 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:17:701.532 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:17:701.581 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:17:701.612 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:17:701.906 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:17:701.995 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:17:702.014 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:17:702.033 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:17:727.983 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:17:728.476 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:17:747.479 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:17:747.631 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:17:747.666 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:17:747.692 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:17:747.722 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:17:748.014 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:17:748.054 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:17:748.075 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:17:748.094 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:47:17:748.112 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:22:588.899 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:22:589.066 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:22:589.093 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:22:589.134 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:22:589.176 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:22:589.214 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:22:589.236 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:22:589.253 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:22:589.266 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:22:589.305 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:22:702.516 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:22:702.644 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:22:702.691 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:22:702.728 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:22:702.866 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:22:702.916 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:22:702.932 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:22:702.947 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:22:728.554 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:22:728.972 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:22:748.089 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:22:748.219 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:22:748.325 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:22:748.354 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:22:748.381 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:22:748.485 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:22:748.605 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:22:748.624 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:22:748.636 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:22:748.649 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:47:27:589.162 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:27:589.375 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:27:589.415 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:27:589.471 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:27:589.514 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:27:589.553 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:27:589.615 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:27:589.640 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:27:589.660 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:27:589.678 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:27:702.948 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:27:703.085 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:27:703.119 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:27:703.173 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:27:703.512 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:27:703.542 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:27:703.555 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:27:703.643 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:27:729.063 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:27:729.548 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:27:748.555 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:27:748.720 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:27:748.753 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:27:748.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:27:748.802 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:27:749.021 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:27:749.057 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:27:749.090 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:27:749.102 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:27:749.114 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:47:32:589.460 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:32:589.670 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:32:589.718 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:32:589.785 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:32:589.804 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:32:589.849 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:32:589.895 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:32:589.944 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:32:589.968 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:32:589.987 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:32:703.617 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:32:703.741 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:32:703.779 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:32:703.812 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:32:703.933 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:32:704.052 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:32:704.080 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:32:704.102 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:32:729.609 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:32:729.996 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:32:749.089 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:32:755.052 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:32:755.104 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:32:755.134 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:32:755.169 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:32:755.368 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:32:755.421 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:32:755.442 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:32:755.466 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:32:755.481 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:47:37:589.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:37:589.989 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:37:590.016 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:37:590.054 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:37:590.093 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:37:590.108 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:37:590.118 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:37:590.161 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:37:590.195 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:37:590.210 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:37:704.024 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:37:704.239 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:37:704.291 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:37:704.333 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:37:704.603 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:37:704.642 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:37:704.721 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:37:704.745 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:37:730.096 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:37:730.650 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:37:755.478 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:37:755.624 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:37:755.662 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:37:755.705 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:37:755.731 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:37:756.127 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:37:756.170 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:37:756.186 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:37:756.201 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:47:37:756.214 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:37:954.058 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-30 Mon 03:47:37:954.121 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:37:954.196 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: SANDESH: Send FAILED: 1774842457953938 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 104 dequeues = 104 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 141 dequeues = 141 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842435471426 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842435471939 ] max_sm_queue_count = 6 ] 2026-03-30 Mon 03:47:37:954.210 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:37:954.220 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:37:954.362 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: SANDESH: Send FAILED: 1774842457954042 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 93, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:47:37:954.391 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719328519936, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:42:590.057 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:42:590.248 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:42:590.280 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:42:590.336 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:42:590.377 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:42:590.415 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:42:590.459 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:42:590.480 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:42:590.497 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:42:590.514 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:42:704.972 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:42:705.091 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:42:705.121 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:42:705.166 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:42:705.454 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:42:705.495 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:42:705.511 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:42:705.526 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:42:730.733 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:42:731.178 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:42:756.208 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:42:756.335 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:42:756.367 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:42:756.396 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:42:756.422 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:42:756.704 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:42:758.360 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:42:758.385 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:42:758.400 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:42:758.415 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:47:45:472.488 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:47:45:472.743 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Connect : Start Connect timer 10.0.0.241:8086 2026-03-30 Mon 03:47:45:472.864 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:47:45:472.894 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: SANDESH: Send FAILED: 1774842465472562 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:47:45:472.911 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:47:45:472.920 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:47:45:473.058 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: SANDESH: Send FAILED: 1774842465472839 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 93, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 75, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:47:45:473.074 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:45:484.719 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:47:45:484.768 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:47:45:484.825 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:47:45:484.842 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:47:45:485.025 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:47:45:485.070 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Idle 2026-03-30 Mon 03:47:45:485.191 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:45:485.428 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: SANDESH: Send FAILED: 1774842465484957 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 93, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 77, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:47:45:485.455 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:45:485.471 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:45:485.504 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: SANDESH: Send FAILED: 1774842465484993 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:47:45:485.515 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:47:45:485.537 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:47:45:485.551 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:45:485.754 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: SANDESH: Send FAILED: 1774842465485165 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 94, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 77, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:47:45:485.772 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:47:590.336 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:47:590.539 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:47:590.572 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:47:590.620 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:47:590.654 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:47:590.683 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:47:590.735 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:47:590.752 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:47:590.765 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:47:590.777 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:47:705.538 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:47:705.698 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:47:705.754 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:47:705.805 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:47:705.899 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:47:705.953 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:47:706.021 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:47:706.214 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:47:731.261 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:47:733.091 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:47:758.394 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:47:758.552 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:47:758.606 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:47:758.647 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:47:758.687 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:47:758.859 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:47:758.927 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:47:758.955 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:47:758.976 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:47:758.998 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:47:52:590.602 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:52:590.734 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:52:590.779 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:52:590.823 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:52:590.851 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:52:590.888 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:52:590.903 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:52:590.916 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:52:590.925 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:52:590.957 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:52:705.982 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:52:706.119 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:52:706.182 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:52:706.257 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:52:706.432 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:52:706.466 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:52:706.500 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:52:706.531 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:52:733.185 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:52:733.573 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:52:758.959 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:52:759.107 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:52:759.176 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:52:759.219 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:52:759.255 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:52:759.501 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:52:759.539 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:52:759.600 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:52:759.621 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:52:759.641 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:47:57:590.958 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:47:57:591.134 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:47:57:591.186 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:47:57:591.236 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a556a0 2026-03-30 Mon 03:47:57:591.269 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54d60 2026-03-30 Mon 03:47:57:591.282 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a54420 2026-03-30 Mon 03:47:57:591.292 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:47:57:591.320 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:47:57:591.351 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a53ae0 2026-03-30 Mon 03:47:57:591.365 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a531a0 2026-03-30 Mon 03:47:57:706.500 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:47:57:706.627 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:47:57:706.667 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:47:57:706.704 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:47:57:706.902 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52340 2026-03-30 Mon 03:47:57:706.965 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a51ca0 2026-03-30 Mon 03:47:57:706.984 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a4b1d0 2026-03-30 Mon 03:47:57:706.997 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a52870 2026-03-30 Mon 03:47:57:733.651 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:47:57:734.107 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a515f0 2026-03-30 Mon 03:47:57:759.588 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:47:57:759.722 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:47:57:759.754 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:47:57:759.814 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:47:57:759.843 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:47:57:760.070 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a55fe0 2026-03-30 Mon 03:47:57:760.106 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a584e0 2026-03-30 Mon 03:47:57:760.126 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57ba0 2026-03-30 Mon 03:47:57:760.145 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a57260 2026-03-30 Mon 03:47:57:760.163 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: -1 0x1a56920 2026-03-30 Mon 03:48:02:591.296 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a556a0 2026-03-30 Mon 03:48:02:591.495 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a556a0 2026-03-30 Mon 03:48:02:592.492 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:592.528 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:592.541 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54d60 2026-03-30 Mon 03:48:02:592.632 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a54420 2026-03-30 Mon 03:48:02:592.670 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a53ae0 2026-03-30 Mon 03:48:02:592.703 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a531a0 2026-03-30 Mon 03:48:02:592.761 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a54d60 2026-03-30 Mon 03:48:02:592.769 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:592.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:592.789 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a54420 2026-03-30 Mon 03:48:02:592.794 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:592.802 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:592.810 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a53ae0 2026-03-30 Mon 03:48:02:592.815 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:592.823 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:592.838 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a531a0 2026-03-30 Mon 03:48:02:592.843 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:592.851 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:593.068 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:593.083 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:593.106 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:593.115 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:593.170 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:593.179 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:593.190 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:593.199 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:593.210 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:593.219 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:593.243 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:593.672 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:02:593.701 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: SANDESH: Send FAILED: 1774842482593271 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.249:6379::Down.10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:02:593.716 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:02:593.734 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:593.752 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:593.767 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:593.782 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:707.009 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52340 2026-03-30 Mon 03:48:02:707.164 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a51ca0 2026-03-30 Mon 03:48:02:707.213 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a4b1d0 2026-03-30 Mon 03:48:02:707.249 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a52870 2026-03-30 Mon 03:48:02:707.342 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a52340 2026-03-30 Mon 03:48:02:707.353 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:707.388 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:707.433 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a4b1d0 2026-03-30 Mon 03:48:02:707.442 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:707.455 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:707.473 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a52870 2026-03-30 Mon 03:48:02:707.480 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:707.491 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:707.881 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a51ca0 2026-03-30 Mon 03:48:02:707.899 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:707.916 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:708.924 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:708.967 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:708.988 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:709.000 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:709.018 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:709.030 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:709.048 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:709.061 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:709.092 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:709.178 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:709.202 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:709.222 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:709.263 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:02:709.325 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: SANDESH: Send FAILED: 1774842482709130 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:02:709.336 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719324321536, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:02:734.181 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a515f0 2026-03-30 Mon 03:48:02:734.581 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a515f0 2026-03-30 Mon 03:48:02:734.611 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:734.638 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:737.395 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:737.429 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:737.454 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:760.151 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a55fe0 2026-03-30 Mon 03:48:02:760.268 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a584e0 2026-03-30 Mon 03:48:02:760.310 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57ba0 2026-03-30 Mon 03:48:02:760.339 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a57260 2026-03-30 Mon 03:48:02:760.367 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1a56920 2026-03-30 Mon 03:48:02:760.609 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a55fe0 2026-03-30 Mon 03:48:02:760.645 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:760.670 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:760.682 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a584e0 2026-03-30 Mon 03:48:02:760.688 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:760.697 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:760.719 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a57260 2026-03-30 Mon 03:48:02:760.725 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:760.734 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:760.742 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a56920 2026-03-30 Mon 03:48:02:760.748 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:760.764 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:760.772 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: RAC_Connect status: 0 0x1a57ba0 2026-03-30 Mon 03:48:02:760.778 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:760.785 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:761.100 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:761.146 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:761.171 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:761.187 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:761.196 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:761.208 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:761.224 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:761.306 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:761.324 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:761.341 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:761.349 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:02:761.351 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:761.409 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: SANDESH: Send FAILED: 1774842482761252 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:02:761.427 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:761.448 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:761.457 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:761.471 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:761.428 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719366305536, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:07:954.796 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:07:954.908 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: SANDESH: Send FAILED: 1774842487954672 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 114 dequeues = 114 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 154 dequeues = 154 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842465485068 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842482761338 ] max_sm_queue_count = 6 ] 2026-03-30 Mon 03:48:07:954.923 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:07:954.934 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:07:955.079 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: SANDESH: Send FAILED: 1774842487954769 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 83, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:48:07:955.109 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:12:959.737 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:12:959.822 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: SANDESH: Send FAILED: 1774842492959705 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.121306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-30 Mon 03:48:12:959.833 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719320123136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-03-30 Mon 03:48:15:487.203 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:48:15:487.390 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Connect : Start Connect timer 10.0.0.249:8086 2026-03-30 Mon 03:48:15:487.481 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:48:15:487.510 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842495487275 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:15:487.527 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:15:487.537 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:48:15:487.697 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842495487470 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 86, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:48:15:487.713 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:15:488.617 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:48:15:488.647 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:48:15:488.681 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:48:15:488.700 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:48:15:488.809 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:48:15:488.839 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Idle 2026-03-30 Mon 03:48:15:488.897 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:15:489.029 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: SANDESH: Send FAILED: 1774842495488761 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 88, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:48:15:489.045 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:15:489.055 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:15:489.076 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: SANDESH: Send FAILED: 1774842495488784 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:15:489.085 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:15:489.094 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:48:15:489.114 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:15:489.247 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: SANDESH: Send FAILED: 1774842495488888 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 155, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 88, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:48:15:489.822 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719357908736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:37:955.427 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719341115136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-30 Mon 03:48:37:955.500 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719341115136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:37:955.568 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719341115136, Pid 1]: SANDESH: Send FAILED: 1774842517955252 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 122 dequeues = 122 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 165 dequeues = 165 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842495488838 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842495489113 ] max_sm_queue_count = 6 ] 2026-03-30 Mon 03:48:37:955.594 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719341115136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:37:955.604 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719341115136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:37:955.747 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719341115136, Pid 1]: SANDESH: Send FAILED: 1774842517955402 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 155, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 91, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:48:37:955.763 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719341115136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:45:490.626 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:48:45:490.804 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Connect : Start Connect timer 10.0.0.141:8086 2026-03-30 Mon 03:48:45:490.895 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:48:45:490.923 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: SANDESH: Send FAILED: 1774842525490694 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:45:490.932 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:45:490.941 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:48:45:491.073 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: SANDESH: Send FAILED: 1774842525490885 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 155, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 93, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:48:45:491.093 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:45:497.149 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:55604::10.0.0.141:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:48:45:497.190 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:55604::10.0.0.141:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:48:45:497.202 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719509977216, Pid 1]: 10.0.0.241:55604::10.0.0.141:8086(-1):EvTcpConnected 2026-03-30 Mon 03:48:45:497.279 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:48:45:497.316 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:48:45:497.358 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: ClientInit 2026-03-30 Mon 03:48:45:497.370 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-2.:contrail-query-engine:0:Database count 7 2026-03-30 Mon 03:48:45:497.455 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:48:45:497.465 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:48:45:497.489 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: SANDESH: Send FAILED: 1774842525497349 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:45:497.497 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:48:45:497.506 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:48:45:497.514 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:48:45:498.324 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:48:45:498.358 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:48:45:498.521 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: SANDESH: Send FAILED: 1774842525497445 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 156, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 95, _iter59->first = sent _iter59->second = 31, ] ] ] 2026-03-30 Mon 03:48:45:498.537 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:48:45:502.222 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: OnMessage control in state: ClientInit 2026-03-30 Mon 03:48:45:502.265 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-30 Mon 03:48:45:502.274 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-30 Mon 03:48:45:502.298 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Received Ctrl Message with size 0 2026-03-30 Mon 03:48:45:502.575 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 204968 calls = 27 average_bytes = 7591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 122 dequeues = 122 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 165 dequeues = 165 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842495488838 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842495489113 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 156, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 95, _iter59->first = sent _iter59->second = 31, ] ] max_sm_queue_count = 6 proxy seq 67 2026-03-30 Mon 03:48:45:502.623 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:48:45:502.647 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 46 2026-03-30 Mon 03:48:45:502.662 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:48:45:502.693 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.121306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 5 2026-03-30 Mon 03:48:45:502.703 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:48:45:502.711 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:48:45:502.741 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Established 2026-03-30 Mon 03:49:37:956.931 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-30 Mon 03:50:37:959.496 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719345313536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-30 Mon 03:51:37:960.079 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-30 Mon 03:52:37:961.155 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719336916736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-03-30 Mon 03:53:14:688.818 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvStop in state Established 2026-03-30 Mon 03:53:14:688.910 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Idle 2026-03-30 Mon 03:53:14:689.007 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:53:14:689.019 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:53:14:689.164 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: SANDESH: Send FAILED: 1774842794688995 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774842212853635 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3051 calls = 6 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 479507 calls = 54 average_bytes = 8879 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 32 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 56 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 156, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 97, _iter59->first = sent _iter59->second = 61, ] ] ] 2026-03-30 Mon 03:53:14:689.226 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140719311726336, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:53:42:802.242 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: http-server-port 8091 2026-03-30 Mon 03:53:42:802.311 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Max-tasks 16 2026-03-30 Mon 03:53:42:802.318 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Max-slice 100 2026-03-30 Mon 03:53:42:802.324 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Collectors 10.0.0.249:8086 2026-03-30 Mon 03:53:42:802.329 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Collectors 10.0.0.241:8086 2026-03-30 Mon 03:53:42:802.333 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Collectors 10.0.0.141:8086 2026-03-30 Mon 03:53:42:802.353 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: ROLE : Generator 2026-03-30 Mon 03:53:42:802.359 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-30 Mon 03:53:42:802.363 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5563-2. 2026-03-30 Mon 03:53:42:802.368 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-30 Mon 03:53:42:802.373 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-30 Mon 03:53:42:802.378 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-30 Mon 03:53:42:802.384 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-30 Mon 03:53:42:805.211 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: HTTP Introspect Init 2026-03-30 Mon 03:53:42:805.319 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 138 2026-03-30 Mon 03:53:42:805.402 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Sandesh Http Server Port: 8091 2026-03-30 Mon 03:53:42:805.455 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-30 Mon 03:53:42:805.463 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-30 Mon 03:53:42:805.777 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Idle 2026-03-30 Mon 03:53:42:805.907 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: No Client: 1774842822805855 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842822805788 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:53:42:806.037 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-30 Mon 03:53:42:806.051 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-30 Mon 03:53:42:806.061 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Cassandra Servers: 10.0.0.249:9042 10.0.0.241:9042 10.0.0.141:9042 2026-03-30 Mon 03:53:42:809.389 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Processing scm::EvStart in state Idle 2026-03-30 Mon 03:53:42:809.468 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Disconnect 2026-03-30 Mon 03:53:42:809.536 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-30 Mon 03:53:42:809.741 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Connect : Start Connect timer 10.0.0.249:8086 2026-03-30 Mon 03:53:42:809.801 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:53:42:809.819 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SANDESH: Send FAILED: 1774842822809461 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:53:42:809.827 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:53:42:809.835 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:53:42:809.899 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SANDESH: Send FAILED: 1774842822809525 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842822805788 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:53:42:809.915 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:53:42:809.923 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:53:42:809.969 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SANDESH: Send FAILED: 1774842822809573 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842822805788 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:53:42:809.978 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:53:42:809.986 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:53:42:809.997 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SANDESH: Send FAILED: 1774842822809589 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:53:42:810.006 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:53:42:810.013 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:53:42:810.051 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SANDESH: Send FAILED: 1774842822809790 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842822805788 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:53:42:810.060 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:53:42:869.144 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-30 Mon 03:53:42:881.186 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb6e4f0 2026-03-30 Mon 03:53:42:881.218 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.233 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.246 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb6f0b0 2026-03-30 Mon 03:53:42:881.251 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.259 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.268 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb6fcf0 2026-03-30 Mon 03:53:42:881.273 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.281 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.290 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb70930 2026-03-30 Mon 03:53:42:881.295 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.302 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.311 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb71570 2026-03-30 Mon 03:53:42:881.316 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.323 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.332 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb66580 2026-03-30 Mon 03:53:42:881.337 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.344 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.353 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb6b540 2026-03-30 Mon 03:53:42:881.358 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.365 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.373 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb6beb0 2026-03-30 Mon 03:53:42:881.378 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.394 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.403 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb6d830 2026-03-30 Mon 03:53:42:881.408 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.415 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.424 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb6cb70 2026-03-30 Mon 03:53:42:881.429 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.436 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.445 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb721b0 2026-03-30 Mon 03:53:42:881.449 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.456 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.465 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb72df0 2026-03-30 Mon 03:53:42:881.469 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.476 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.484 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb74670 2026-03-30 Mon 03:53:42:881.489 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.495 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.503 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb73a30 2026-03-30 Mon 03:53:42:881.508 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.514 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:881.523 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: RAC_Connect status: 0 0xb752b0 2026-03-30 Mon 03:53:42:881.528 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:53:42:881.534 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:53:42:882.945 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:54214::10.0.0.249:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2026-03-30 Mon 03:53:42:882.965 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: 10.0.0.241:54214::10.0.0.249:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:53:42:882.972 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: 10.0.0.241:54214::10.0.0.249:8086(-1):EvTcpConnected 2026-03-30 Mon 03:53:42:883.156 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:53:42:883.195 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: SANDESH: Send FAILED: 1774842822881072 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:53:42:883.203 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:53:42:883.213 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:53:42:883.220 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:53:42:883.251 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: ClientInit 2026-03-30 Mon 03:53:42:883.265 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-2.:contrail-query-engine:0:Database count 1 2026-03-30 Mon 03:53:42:883.340 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:53:42:883.348 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:53:42:883.362 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: SANDESH: Send FAILED: 1774842822883246 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:53:42:883.368 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:53:42:883.375 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:53:42:883.381 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:53:42:883.403 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:53:42:883.409 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:53:42:883.490 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: SANDESH: Send FAILED: 1774842822883329 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774842822805788 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:53:42:883.504 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:53:42:885.358 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.387 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.400 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.409 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.421 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.429 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.439 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.446 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.456 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.464 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.511 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:885.533 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:885.545 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:885.565 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:885.576 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:885.635 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:53:42:885.645 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:53:42:885.664 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: SANDESH: Send FAILED: 1774842822885499 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:53:42:885.671 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:53:42:885.678 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:53:42:885.684 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:53:42:885.696 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: SANDESH: Send FAILED: 1774842822885528 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.249:6379::Down.10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:53:42:885.701 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:53:42:885.767 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.795 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.836 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:885.858 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.867 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.879 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.887 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.898 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.907 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.917 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.924 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.935 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.942 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.952 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.960 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.970 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.977 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:885.988 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:885.996 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:886.038 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:53:42:886.041 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:886.086 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:886.097 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:886.106 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:886.116 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:886.126 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:886.135 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:886.144 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:886.163 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:53:42:886.179 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:53:42:886.205 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:53:42:886.062 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:53:42:886.262 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SANDESH: Send FAILED: 1774842822886023 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:53:42:886.270 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:53:42:886.280 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:53:42:886.286 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:53:42:886.299 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SANDESH: Send FAILED: 1774842822886081 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:53:42:886.310 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:53:42:888.779 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: OnMessage control in state: ClientInit 2026-03-30 Mon 03:53:42:888.834 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-30 Mon 03:53:42:888.842 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-30 Mon 03:53:42:888.860 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Received Ctrl Message with size 0 2026-03-30 Mon 03:53:42:888.958 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774842822805788 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2026-03-30 Mon 03:53:42:888.984 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:53:42:889.004 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2026-03-30 Mon 03:53:42:889.016 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:53:42:889.023 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:53:42:889.029 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:53:42:889.061 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: Established 2026-03-30 Mon 03:54:42:806.375 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337909581376, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-30 Mon 03:55:42:807.123 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-30 Mon 03:56:42:807.717 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337917978176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-30 Mon 03:57:42:808.196 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337917978176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-30 Mon 03:58:42:808.963 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337917978176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-30 Mon 03:59:42:809.538 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338714916416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-30 Mon 04:00:42:810.051 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-30 Mon 04:01:42:810.655 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-30 Mon 04:02:42:811.534 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338672670272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-03-30 Mon 04:03:42:812.099 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337909581376, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2026-03-30 Mon 04:04:42:812.913 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337909581376, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2026-03-30 Mon 04:05:42:813.446 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2026-03-30 Mon 04:06:42:813.942 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337917978176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2026-03-30 Mon 04:07:42:814.478 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338668471872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2026-03-30 Mon 04:08:42:815.107 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2026-03-30 Mon 04:09:42:815.687 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337909581376, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2026-03-30 Mon 04:10:42:816.234 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338714916416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2026-03-30 Mon 04:11:33:398.701 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:8091::10.0.0.241:54774 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-30 Mon 04:11:42:816.761 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338681067072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2026-03-30 Mon 04:11:52:858.954 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:8091::10.0.0.241:54022 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-30 Mon 04:12:42:817.273 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337917978176, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2026-03-30 Mon 04:13:33:354.060 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:8091::10.0.0.249:40340 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-30 Mon 04:13:42:817.741 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2026-03-30 Mon 04:13:57:797.786 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.249 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-03-30 Mon 04:13:57:798.497 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID df8c8ef2-2bee-11f1-a24f-00000a0000f9 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-03-30 Mon 04:13:57:798.550 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QEQueryLog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 :0:json query is: { end_time: 1774844037620493 , enqueue_time: 1774844037797495 , query_metadata: {"enqueue_time": 1774844037797326} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "301c349e-4d87-4414-8c1f-0ca1dc98e33c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:13:57:798.709 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-03-30 Mon 04:13:57:798.725 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-03-30 Mon 04:13:57:798.769 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for df8c8ef2-2bee-11f1-a24f-00000a0000f9 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-03-30 Mon 04:13:57:798.885 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338706519616, Pid 1]: [SYS_INFO]: QEQueryLog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 :0:json query is: { end_time: 1774844037620493 , enqueue_time: 1774844037797495 , query_metadata: {"enqueue_time": 1774844037797326} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "301c349e-4d87-4414-8c1f-0ca1dc98e33c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:13:57:863.793 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:863.834 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:863.846 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:863.857 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:863.867 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:863.877 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:863.887 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:863.897 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:863.907 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:863.918 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:865.637 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:865.665 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:865.677 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:865.716 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:865.740 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:865.757 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:865.769 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:865.783 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:865.794 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:865.808 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:866.118 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QEQueryLog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 301c349e-4d87-4414-8c1f-0ca1dc98e33c src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:57:866.307 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QEQueryLog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 301c349e-4d87-4414-8c1f-0ca1dc98e33c src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:57:866.490 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QEQueryLog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 301c349e-4d87-4414-8c1f-0ca1dc98e33c src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:57:866.666 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QEQueryLog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 301c349e-4d87-4414-8c1f-0ca1dc98e33c src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:57:866.809 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337905382976, Pid 1]: [SYS_INFO]: QEQueryLog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 301c349e-4d87-4414-8c1f-0ca1dc98e33c src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:57:874.916 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:874.945 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:874.953 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:874.971 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:874.977 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:874.984 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:874.990 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:874.996 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:875.003 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:57:875.009 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:57:875.086 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QEQueryLog: df8c8ef2-2bee-11f1-a24f-00000a0000f9 :0:json query is: { end_time: 1774844037620493 , enqueue_time: 1774844037797495 , query_metadata: {"enqueue_time": 1774844037797326} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "301c349e-4d87-4414-8c1f-0ca1dc98e33c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:13:57:875.345 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-03-30 Mon 04:13:57:875.460 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-03-30 Mon 04:13:57:875.521 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-03-30 Mon 04:13:57:876.059 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338706519616, Pid 1]: [SYS_INFO]: QELog: Finished: QID df8c8ef2-2bee-11f1-a24f-00000a0000f9 Table ObjectVMTable Time(ms) 77 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-03-30 Mon 04:13:57:876.102 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338730202688, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5563-2. table = ObjectVMTable query_stats= [ qid = df8c8ef2-2bee-11f1-a24f-00000a0000f9 where = [[{"name": "ObjectId", "value": "301c349e-4d87-4414-8c1f-0ca1dc98e33c", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 66,0, 64,0, 75,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 77 rows = 5 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-03-30 Mon 04:13:57:876.133 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338706519616, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-03-30 Mon 04:14:00:911.050 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.249 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-03-30 Mon 04:14:00:911.841 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID e1675331-2bee-11f1-b61d-00000a0000f9 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-03-30 Mon 04:14:00:911.877 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QEQueryLog: e1675331-2bee-11f1-b61d-00000a0000f9 :0:json query is: { end_time: 1774844040886240 , enqueue_time: 1774844040910459 , query_metadata: {"enqueue_time": 1774844040910158} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482:ctest-vn22-57996482", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:14:00:912.005 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-03-30 Mon 04:14:00:912.020 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-03-30 Mon 04:14:00:912.155 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338668471872, Pid 1]: [SYS_INFO]: QEQueryLog: e1675331-2bee-11f1-b61d-00000a0000f9 :0:json query is: { end_time: 1774844040886240 , enqueue_time: 1774844040910459 , query_metadata: {"enqueue_time": 1774844040910158} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482:ctest-vn22-57996482", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:14:00:912.936 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for e1675331-2bee-11f1-b61d-00000a0000f9 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-03-30 Mon 04:14:00:972.605 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:972.650 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:972.659 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:972.667 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:972.674 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:972.681 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:972.688 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:972.695 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:972.702 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:972.712 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:972.944 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QEQueryLog: e1675331-2bee-11f1-b61d-00000a0000f9 ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:14:00:973.353 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:973.379 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:973.387 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:973.399 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:973.389 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:973.408 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:973.445 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:973.455 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:973.466 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:973.477 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:973.489 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:973.497 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:973.512 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:973.520 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:973.536 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:973.544 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:973.560 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:973.588 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:973.598 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:14:00:973.608 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: [SYS_INFO]: QELog: e1675331-2bee-11f1-b61d-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:14:00:973.680 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337913779776, Pid 1]: [SYS_INFO]: QEQueryLog: e1675331-2bee-11f1-b61d-00000a0000f9 :0:json query is: { end_time: 1774844040886240 , enqueue_time: 1774844040910459 , query_metadata: {"enqueue_time": 1774844040910158} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482:ctest-vn22-57996482", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:14:00:973.917 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-03-30 Mon 04:14:00:973.949 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-03-30 Mon 04:14:00:974.031 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338676868672, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-03-30 Mon 04:14:00:974.581 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: Finished: QID e1675331-2bee-11f1-b61d-00000a0000f9 Table ObjectRoutingInstance Time(ms) 62 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-03-30 Mon 04:14:00:974.608 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338710718016, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5563-2. table = ObjectRoutingInstance query_stats= [ qid = e1675331-2bee-11f1-b61d-00000a0000f9 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482:ctest-vn22-57996482", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 60,0, 59,0, 59,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 62 rows = 1 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-03-30 Mon 04:14:00:974.663 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-03-30 Mon 04:14:42:818.443 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337909581376, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2026-03-30 Mon 04:15:42:818.992 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337922176576, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2026-03-30 Mon 04:16:42:820.040 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337901184576, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2026-03-30 Mon 04:17:42:820.865 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338734401088, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2026-03-30 Mon 04:18:42:821.381 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337909581376, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2026-03-30 Mon 04:19:42:821.875 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338714916416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2026-03-30 Mon 04:20:42:822.358 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338714916416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2026-03-30 Mon 04:21:42:822.989 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338706519616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2026-03-30 Mon 04:22:42:823.609 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338714916416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2026-03-30 Mon 04:23:42:824.117 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140337909581376, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2026-03-30 Mon 04:24:42:824.779 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338706519616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2026-03-30 Mon 04:25:39:810.978 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:8091::10.0.0.249:49142 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-30 Mon 04:25:42:825.472 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338734401088, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2026-03-30 Mon 04:26:42:825.997 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338668471872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2026-03-30 Mon 04:27:36:020.715 UTC cn-jenkins-deploy-platform-ansible-os-5563-2 [Thread 140338741540992, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.241:8091::10.0.0.241:43200 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256