2026-03-23 Mon 05:37:11:011.118 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: http-server-port 8091 2026-03-23 Mon 05:37:11:011.204 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Max-tasks 16 2026-03-23 Mon 05:37:11:011.213 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Max-slice 100 2026-03-23 Mon 05:37:11:011.219 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Collectors 10.0.0.38:8086 2026-03-23 Mon 05:37:11:011.224 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Collectors 10.0.0.50:8086 2026-03-23 Mon 05:37:11:011.229 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Collectors 10.0.0.33:8086 2026-03-23 Mon 05:37:11:011.248 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: ROLE : Generator 2026-03-23 Mon 05:37:11:011.254 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-23 Mon 05:37:11:011.259 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5515-3. 2026-03-23 Mon 05:37:11:011.265 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-23 Mon 05:37:11:011.270 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-23 Mon 05:37:11:011.276 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-23 Mon 05:37:11:011.282 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-23 Mon 05:37:11:011.803 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: HTTP Introspect Init 2026-03-23 Mon 05:37:11:011.913 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-03-23 Mon 05:37:11:011.970 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Sandesh Http Server Port: 8091 2026-03-23 Mon 05:37:11:012.008 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-23 Mon 05:37:11:012.017 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-23 Mon 05:37:11:012.090 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Idle 2026-03-23 Mon 05:37:11:012.242 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: No Client: 1774244231012169 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244231012099 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:11:012.372 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-23 Mon 05:37:11:012.386 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-23 Mon 05:37:11:012.398 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-03-23 Mon 05:37:11:012.621 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvStart in state Idle 2026-03-23 Mon 05:37:11:012.690 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Disconnect 2026-03-23 Mon 05:37:11:012.761 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-23 Mon 05:37:11:012.971 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-23 Mon 05:37:11:016.947 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:11:016.991 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: SANDESH: Send FAILED: 1774244231012681 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:11:017.001 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:11:017.011 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:11:017.113 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: SANDESH: Send FAILED: 1774244231012751 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244231012099 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:11:017.138 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:11:017.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:11:017.231 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: SANDESH: Send FAILED: 1774244231012809 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244231012099 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:11:017.244 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:11:017.254 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:11:017.269 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: SANDESH: Send FAILED: 1774244231012829 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:11:017.275 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:11:017.283 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:11:017.345 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: SANDESH: Send FAILED: 1774244231015304 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244231012099 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:11:017.356 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:11:022.898 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:11:022.933 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:11:022.996 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:11:023.051 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910664500992, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:11:023.081 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910664500992, Pid 1]: SANDESH: Send FAILED: 1774244231022969 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:11:023.090 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910664500992, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:16:025.445 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:16:025.485 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:16:025.521 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:21:027.101 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:21:027.140 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:21:027.162 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:26:028.685 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:26:028.728 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:26:028.748 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:31:272.449 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-23 Mon 05:37:31:272.508 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:31:272.531 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:36:304.948 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-23 Mon 05:37:36:305.000 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:36:305.023 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:41:368.752 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:41:368.806 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:46:399.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-23 Mon 05:37:51:403.913 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:51:403.993 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: SANDESH: Send FAILED: 1774244271403838 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:51:404.001 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:37:51:404.042 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:37:51:404.068 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x1256d80 2026-03-23 Mon 05:37:51:404.077 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.095 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.111 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x1258330 2026-03-23 Mon 05:37:51:404.119 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.132 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.145 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x1250250 2026-03-23 Mon 05:37:51:404.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.166 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.179 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x1258c70 2026-03-23 Mon 05:37:51:404.187 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.198 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.211 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x12565f0 2026-03-23 Mon 05:37:51:404.219 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.249 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.262 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x12595b0 2026-03-23 Mon 05:37:51:404.270 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.283 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.305 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x125b170 2026-03-23 Mon 05:37:51:404.314 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.327 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.340 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x1259ef0 2026-03-23 Mon 05:37:51:404.348 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.361 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.374 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x1257310 2026-03-23 Mon 05:37:51:404.382 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.395 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.408 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x125bab0 2026-03-23 Mon 05:37:51:404.415 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.449 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.463 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x125c3f0 2026-03-23 Mon 05:37:51:404.470 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.482 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.495 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x125a830 2026-03-23 Mon 05:37:51:404.503 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.515 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.528 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x12579f0 2026-03-23 Mon 05:37:51:404.536 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.548 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.561 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x125cd30 2026-03-23 Mon 05:37:51:404.582 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.004 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:404.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:37:51:404.626 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:37:51:404.603 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.656 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: 0 0x125d670 2026-03-23 Mon 05:37:51:404.664 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:404.687 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:404.746 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:37:51:404.790 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Idle 2026-03-23 Mon 05:37:51:404.850 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:404.970 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: SANDESH: Send FAILED: 1774244271404698 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244231012099 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 35 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 36, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:51:404.988 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:404.999 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:405.024 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: SANDESH: Send FAILED: 1774244271404723 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:51:405.031 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:405.039 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:37:51:405.073 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:405.175 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:405.251 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:405.274 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:405.297 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:405.317 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:405.332 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:405.187 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: SANDESH: Send FAILED: 1774244271404841 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244231012099 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 36 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 38, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:51:405.357 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:405.386 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:405.452 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:405.496 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:405.366 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:405.532 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:405.537 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:405.556 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: SANDESH: Send FAILED: 1774244271405439 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:37:51:405.565 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:405.569 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:405.611 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:405.582 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:405.630 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:405.650 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: SANDESH: Send FAILED: 1774244271405486 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:51:405.658 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910933690112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:405.664 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:405.682 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:405.708 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:405.727 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:405.751 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:405.768 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:405.789 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:405.808 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:405.827 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:405.847 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:405.894 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:405.931 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:405.973 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:405.999 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:406.022 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:406.046 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:406.073 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:406.088 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:406.106 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:406.120 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:406.139 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:406.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:406.171 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:406.185 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:406.208 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:405.790 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:406.226 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:406.307 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: SANDESH: Send FAILED: 1774244271405690 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244271404789 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244271405580 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:37:51:406.320 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:406.324 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:406.375 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:406.397 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:406.417 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:406.438 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:406.335 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:406.725 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: SANDESH: Send FAILED: 1774244271405747 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244231012099 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 54 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 56, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:51:406.742 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:406.753 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:406.774 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: SANDESH: Send FAILED: 1774244271405962 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:51:406.784 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:406.795 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:406.813 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: SANDESH: Send FAILED: 1774244271406364 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:51:406.819 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:52:404.933 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:37:52:405.135 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:37:52:405.240 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:52:405.271 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: SANDESH: Send FAILED: 1774244272405003 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:52:405.280 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:52:405.290 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:52:405.409 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: SANDESH: Send FAILED: 1774244272405227 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244231012099 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:52:405.432 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:52:407.170 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:34170::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:37:52:407.200 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: 10.0.0.33:34170::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:37:52:407.209 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: 10.0.0.33:34170::10.0.0.50:8086(-1):EvTcpConnected 2026-03-23 Mon 05:37:52:407.275 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:37:52:407.305 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:37:52:407.338 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: ClientInit 2026-03-23 Mon 05:37:52:407.358 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-3.:contrail-query-engine:0:Database count 1 2026-03-23 Mon 05:37:52:407.439 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:37:52:407.449 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:37:52:407.471 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: SANDESH: Send FAILED: 1774244272407331 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:52:407.478 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:37:52:407.488 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:37:52:407.494 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:37:52:407.506 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:37:52:407.512 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:37:52:407.618 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: SANDESH: Send FAILED: 1774244272407428 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244231012099 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:52:407.636 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:37:52:411.480 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:37:52:411.533 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:37:52:411.543 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:37:52:411.567 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:37:52:411.702 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244231012099 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 16 dequeues = 16 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244271404789 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244271405580 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2026-03-23 Mon 05:37:52:411.728 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:37:52:411.754 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2026-03-23 Mon 05:37:52:411.767 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:37:52:411.777 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:37:52:411.785 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:37:52:411.814 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910672897792, Pid 1]: Established 2026-03-23 Mon 05:38:00:038.116 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: OnMessage control in state: Established 2026-03-23 Mon 05:38:00:038.175 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-03-23 Mon 05:38:00:038.186 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: Established : EvSandeshMessageRecv 2026-03-23 Mon 05:38:00:038.229 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: Received Ctrl Message with size 3 2026-03-23 Mon 05:38:00:038.242 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2026-03-23 Mon 05:38:00:038.254 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2026-03-23 Mon 05:38:00:038.265 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2026-03-23 Mon 05:38:00:038.273 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910668699392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:38:21:408.267 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910690354944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-23 Mon 05:39:21:410.106 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910912698112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-23 Mon 05:40:21:411.238 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910912698112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-23 Mon 05:40:44:801.941 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.33:49220 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-23 Mon 05:41:16:297.541 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.33:35588 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-23 Mon 05:41:21:413.165 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910664500992, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-23 Mon 05:42:21:414.171 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910921094912, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-23 Mon 05:43:14:841.289 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Disconnect status: -1 0x125a830 2026-03-23 Mon 05:43:14:841.346 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.465 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Disconnect status: -1 0x1259ef0 2026-03-23 Mon 05:43:14:841.479 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.509 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Disconnect status: -1 0x12595b0 2026-03-23 Mon 05:43:14:841.519 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.540 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Disconnect status: -1 0x1258c70 2026-03-23 Mon 05:43:14:841.550 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.572 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:43:14:841.579 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Disconnect status: -1 0x1258330 2026-03-23 Mon 05:43:14:841.588 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:842.973 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: -1 0x125a830 2026-03-23 Mon 05:43:14:843.035 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: -1 0x1259ef0 2026-03-23 Mon 05:43:14:843.052 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: -1 0x12595b0 2026-03-23 Mon 05:43:14:843.065 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: -1 0x1258c70 2026-03-23 Mon 05:43:14:843.080 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139911081560192, Pid 1]: RAC_Connect status: -1 0x1258330 2026-03-23 Mon 05:43:14:862.179 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910702950144, Pid 1]: Processing scm::EvStop in state Established 2026-03-23 Mon 05:43:14:862.288 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910702950144, Pid 1]: Idle 2026-03-23 Mon 05:43:14:862.395 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910702950144, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:43:14:862.407 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910702950144, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:43:14:862.580 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910702950144, Pid 1]: SANDESH: Send FAILED: 1774244594862377 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244231012099 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 286229 calls = 32 average_bytes = 8944 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 22 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 33, ] ] ] 2026-03-23 Mon 05:43:14:862.603 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 139910702950144, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:27:789.768 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: http-server-port 8091 2026-03-23 Mon 05:43:27:789.845 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Max-tasks 16 2026-03-23 Mon 05:43:27:789.854 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Max-slice 100 2026-03-23 Mon 05:43:27:789.860 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Collectors 10.0.0.38:8086 2026-03-23 Mon 05:43:27:789.865 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Collectors 10.0.0.50:8086 2026-03-23 Mon 05:43:27:789.870 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Collectors 10.0.0.33:8086 2026-03-23 Mon 05:43:27:789.888 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: ROLE : Generator 2026-03-23 Mon 05:43:27:789.906 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-23 Mon 05:43:27:789.912 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5515-3. 2026-03-23 Mon 05:43:27:789.917 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-23 Mon 05:43:27:789.923 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-23 Mon 05:43:27:789.928 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-23 Mon 05:43:27:789.934 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-23 Mon 05:43:27:790.506 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: HTTP Introspect Init 2026-03-23 Mon 05:43:27:794.484 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-03-23 Mon 05:43:27:794.566 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Sandesh Http Server Port: 8091 2026-03-23 Mon 05:43:27:794.601 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-23 Mon 05:43:27:794.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-23 Mon 05:43:27:794.707 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Idle 2026-03-23 Mon 05:43:27:794.870 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: No Client: 1774244607794796 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:43:27:795.008 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-23 Mon 05:43:27:795.023 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-23 Mon 05:43:27:795.036 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-03-23 Mon 05:43:27:795.119 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvStart in state Idle 2026-03-23 Mon 05:43:27:795.190 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Disconnect 2026-03-23 Mon 05:43:27:795.260 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-23 Mon 05:43:27:795.562 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:43:27:795.643 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:27:795.704 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244607795182 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:43:27:795.722 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:43:27:795.732 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:27:795.816 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244607795250 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:43:27:795.829 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:27:795.839 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:27:795.929 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244607795307 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:43:27:795.941 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:27:795.951 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:27:795.966 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244607795327 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:43:27:795.973 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:43:27:795.984 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:27:796.046 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244607795630 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:43:27:796.056 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:27:845.179 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-23 Mon 05:43:57:874.161 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:57:874.227 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: SANDESH: Send FAILED: 1774244637874073 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:43:57:874.245 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:43:57:877.007 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:50684::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:43:57:877.038 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:50684::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:43:57:877.047 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:50684::10.0.0.38:8086(-1):EvTcpConnected 2026-03-23 Mon 05:43:57:877.088 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f45f0 2026-03-23 Mon 05:43:57:877.095 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:877.112 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:877.127 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f4ca0 2026-03-23 Mon 05:43:57:877.132 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:877.141 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:877.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28ee1d0 2026-03-23 Mon 05:43:57:877.158 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:877.167 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:877.176 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f5340 2026-03-23 Mon 05:43:57:877.182 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:877.189 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:877.210 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:43:57:877.248 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:43:57:877.262 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f5870 2026-03-23 Mon 05:43:57:877.267 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:877.276 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:877.293 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:43:57:877.309 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:43:57:877.324 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:43:57:877.338 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:43:57:877.351 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:43:57:877.366 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:43:57:877.381 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:43:57:877.396 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:43:57:877.549 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:43:57:878.479 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:43:57:878.550 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: ClientInit 2026-03-23 Mon 05:43:57:878.571 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-3.:contrail-query-engine:0:Database count 1 2026-03-23 Mon 05:43:57:878.671 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:878.681 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:43:57:878.702 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: SANDESH: Send FAILED: 1774244637878540 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:43:57:878.711 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:43:57:878.719 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:878.726 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:43:57:878.737 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:878.744 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:43:57:878.853 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: SANDESH: Send FAILED: 1774244637878659 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 8, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:43:57:878.873 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:43:57:879.732 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:879.751 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:43:57:879.810 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: SANDESH: Send FAILED: 1774244637879598 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1774244637878549 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1774244637878736 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:43:57:879.821 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:43:57:879.830 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:879.836 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:43:57:879.946 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: SANDESH: Send FAILED: 1774244637879650 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 8, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 8, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:43:57:879.960 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:43:57:881.141 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:881.281 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:881.301 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:881.328 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:881.347 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:881.357 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:881.370 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:881.379 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:881.392 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:881.401 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:881.458 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:881.485 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:881.501 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:881.517 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:881.532 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:881.580 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:881.590 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:43:57:881.612 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: SANDESH: Send FAILED: 1774244637881441 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:43:57:881.623 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:43:57:881.632 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:881.639 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:43:57:881.656 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: SANDESH: Send FAILED: 1774244637881479 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:43:57:881.662 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:43:57:883.097 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:43:57:883.139 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:43:57:883.150 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:43:57:883.174 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:43:57:883.337 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1774244637878549 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1774244637878736 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 0 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 8, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 8, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 7 2026-03-23 Mon 05:43:57:883.365 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:43:57:883.390 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 6 2026-03-23 Mon 05:43:57:883.402 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:43:57:883.410 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:43:57:883.417 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:43:57:883.456 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: Established 2026-03-23 Mon 05:44:02:877.264 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:44:02:877.379 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:44:02:877.414 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:44:02:877.499 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:44:02:877.530 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:44:02:877.558 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:44:02:877.614 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:44:02:877.664 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f61a0 2026-03-23 Mon 05:44:02:877.671 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:44:02:877.690 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:44:02:877.708 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f6ae0 2026-03-23 Mon 05:44:02:877.727 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:44:02:877.736 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:44:02:877.745 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f8fe0 2026-03-23 Mon 05:44:02:877.750 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:44:02:877.758 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:44:02:877.766 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f7420 2026-03-23 Mon 05:44:02:877.771 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:44:02:877.779 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:44:02:877.787 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f7d60 2026-03-23 Mon 05:44:02:877.792 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:44:02:877.799 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:44:02:877.807 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f9920 2026-03-23 Mon 05:44:02:877.812 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:44:02:877.820 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:44:02:877.826 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:44:02:877.862 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:44:02:877.907 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:44:02:877.986 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f86a0 2026-03-23 Mon 05:44:02:877.994 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:44:02:878.005 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:44:02:878.058 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28fa260 2026-03-23 Mon 05:44:02:878.065 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:44:02:878.088 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:44:02:878.112 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28faba0 2026-03-23 Mon 05:44:02:878.119 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:44:02:878.136 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:44:02:878.154 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28fb4e0 2026-03-23 Mon 05:44:02:878.160 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:44:02:878.178 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:44:02:878.211 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:44:02:878.222 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:44:02:878.236 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:44:02:878.245 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:44:02:878.258 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:44:02:878.267 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:44:02:878.279 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:44:02:878.289 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:44:02:878.341 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:44:02:878.413 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:44:02:878.433 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:44:02:878.448 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:44:02:878.471 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:44:02:878.486 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:44:02:878.502 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:44:02:878.930 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:44:02:878.965 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:44:02:878.980 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:44:02:878.990 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:44:02:879.002 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:44:02:879.012 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:44:02:879.024 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:44:02:879.034 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:44:02:879.046 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:44:02:879.056 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:44:02:879.085 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:44:02:879.116 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:44:02:879.131 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:44:02:879.144 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:44:02:879.157 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:44:27:879.869 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102219052800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-23 Mon 05:45:27:880.935 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102214854400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-23 Mon 05:45:55:846.375 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28f86a0 2026-03-23 Mon 05:45:55:846.438 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.479 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28f7d60 2026-03-23 Mon 05:45:55:846.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.513 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28f7420 2026-03-23 Mon 05:45:55:846.524 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.710 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28f6ae0 2026-03-23 Mon 05:45:55:846.731 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.763 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:45:55:846.771 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28f61a0 2026-03-23 Mon 05:45:55:846.785 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.992 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:45:55:847.034 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:45:55:847.050 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:45:55:847.073 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:45:55:847.086 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:45:55:886.231 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28f5870 2026-03-23 Mon 05:45:55:886.349 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.391 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28f5340 2026-03-23 Mon 05:45:55:886.403 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.435 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28ee1d0 2026-03-23 Mon 05:45:55:886.445 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.466 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28f4ca0 2026-03-23 Mon 05:45:55:886.475 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.498 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:45:55:886.507 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28f45f0 2026-03-23 Mon 05:45:55:886.521 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.972 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:45:55:887.003 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:45:55:887.017 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:45:55:887.042 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:45:55:887.116 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:45:55:898.827 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.33:50684::10.0.0.38:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:45:55:898.878 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:50684::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-23 Mon 05:45:55:898.887 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:50684::10.0.0.38:8086(-1):EvTcpClose 2026-03-23 Mon 05:45:55:899.051 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvTcpClose in state Established 2026-03-23 Mon 05:45:55:899.084 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Established : EvTcpClose 2026-03-23 Mon 05:45:55:899.393 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:45:55:899.520 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:55:899.786 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244755899194 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = cn-jenkins-deploy-platform-ansible-os-5515-1. collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 123964 calls = 17 average_bytes = 7292 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 76, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 18, ] ] ] 2026-03-23 Mon 05:45:55:899.815 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:55:899.832 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:55:899.866 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244755899234 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.50:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:45:55:899.878 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:45:55:899.903 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2026-03-23 Mon 05:45:55:899.945 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:55:900.277 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244755899499 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 123964 calls = 17 average_bytes = 7292 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 76, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 18, ] ] ] 2026-03-23 Mon 05:45:55:900.307 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:55:906.546 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:50974::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:45:55:907.698 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:50974::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:45:55:907.718 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:50974::10.0.0.50:8086(-1):EvTcpConnected 2026-03-23 Mon 05:45:55:907.802 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:45:55:907.822 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:45:55:907.872 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: ClientInit 2026-03-23 Mon 05:45:55:907.916 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-3.:contrail-query-engine:0:Database count 2 2026-03-23 Mon 05:45:55:908.046 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:55:908.068 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:45:55:908.103 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: SANDESH: Send FAILED: 1774244755907859 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.50:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:45:55:908.114 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:45:55:908.127 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:55:908.137 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:45:55:908.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:55:908.163 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:45:55:908.411 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: SANDESH: Send FAILED: 1774244755908029 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 123964 calls = 17 average_bytes = 7292 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 73 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 77, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 18, ] ] ] 2026-03-23 Mon 05:45:55:908.466 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:45:55:956.018 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28fb4e0 2026-03-23 Mon 05:45:55:956.058 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:956.167 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28faba0 2026-03-23 Mon 05:45:55:956.182 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:956.208 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28fa260 2026-03-23 Mon 05:45:55:956.218 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:956.241 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28f9920 2026-03-23 Mon 05:45:55:956.251 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:956.276 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:45:55:956.283 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Disconnect status: -1 0x28f8fe0 2026-03-23 Mon 05:45:55:956.291 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:956.517 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:45:55:956.541 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:45:55:956.554 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:45:55:956.566 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:45:55:956.578 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:45:55:956.621 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:55:956.633 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:45:55:956.657 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: SANDESH: Send FAILED: 1774244755956337 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:55:956.672 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:45:55:981.022 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:50974::10.0.0.50:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-23 Mon 05:45:55:981.056 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:50974::10.0.0.50:8086(-1):EvTcpClose 2026-03-23 Mon 05:45:55:981.111 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-23 Mon 05:45:55:981.121 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: ClientInit : EvTcpClose 2026-03-23 Mon 05:45:55:981.379 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Return to idle with 5000 2026-03-23 Mon 05:45:55:981.390 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Idle 2026-03-23 Mon 05:45:55:981.456 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:55:981.615 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: SANDESH: Send FAILED: 1774244755981335 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 125537 calls = 18 average_bytes = 6974 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-03-23 Mon 05:45:55:981.636 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:55:981.647 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:55:981.669 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: SANDESH: Send FAILED: 1774244755981365 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:55:981.676 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:45:55:981.685 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:45:55:981.716 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:55:981.960 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: SANDESH: Send FAILED: 1774244755981442 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 125537 calls = 18 average_bytes = 6974 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-03-23 Mon 05:45:55:981.979 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102309660416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:57:881.502 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:57:881.594 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244757881356 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 40 dequeues = 40 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 49 dequeues = 49 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244755981389 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244755981715 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:45:57:881.607 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:57:881.616 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:57:881.752 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244757881472 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 125537 calls = 18 average_bytes = 6974 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 21, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-03-23 Mon 05:45:57:881.779 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:00:847.089 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:00:847.255 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:00:847.559 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:00:847.595 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:00:847.651 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:00:847.689 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:00:847.737 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:00:847.791 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:00:847.812 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:00:847.830 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:00:887.048 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:00:887.183 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:00:887.223 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:00:887.251 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:00:887.279 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:00:887.518 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:00:887.577 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:00:887.593 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:00:887.621 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:00:887.634 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:00:956.602 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:00:956.811 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:00:956.876 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:00:956.964 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:00:957.012 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:00:957.033 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:00:957.049 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:00:957.097 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:00:957.272 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:00:957.299 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:00:982.078 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:00:982.342 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-23 Mon 05:46:00:982.436 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:00:982.465 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244760982152 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:00:982.475 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:00:982.484 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:00:982.688 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244760982424 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 125537 calls = 18 average_bytes = 6974 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-03-23 Mon 05:46:00:982.731 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:00:984.563 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:48296::10.0.0.33:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:46:00:984.606 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:48296::10.0.0.33:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:46:00:984.615 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:48296::10.0.0.33:8086(-1):EvTcpConnected 2026-03-23 Mon 05:46:00:985.088 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:46:00:985.110 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:46:00:985.151 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: ClientInit 2026-03-23 Mon 05:46:00:985.165 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-3.:contrail-query-engine:0:Database count 3 2026-03-23 Mon 05:46:00:985.265 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:00:985.276 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:46:00:985.301 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: SANDESH: Send FAILED: 1774244760985143 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:00:985.318 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:46:00:985.329 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:00:985.336 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:46:00:985.348 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:00:985.355 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:46:00:985.493 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: SANDESH: Send FAILED: 1774244760985253 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 125537 calls = 18 average_bytes = 6974 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-03-23 Mon 05:46:00:985.513 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:46:00:986.572 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:48296::10.0.0.33:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-23 Mon 05:46:00:986.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:48296::10.0.0.33:8086(-1):EvTcpClose 2026-03-23 Mon 05:46:00:986.856 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-23 Mon 05:46:00:986.943 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: ClientInit : EvTcpClose 2026-03-23 Mon 05:46:00:987.095 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Return to idle with 5000 2026-03-23 Mon 05:46:00:987.106 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Idle 2026-03-23 Mon 05:46:00:987.168 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:00:987.318 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244760987051 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 127110 calls = 19 average_bytes = 6690 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-03-23 Mon 05:46:00:987.339 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:00:987.350 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:00:987.374 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244760987081 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:00:987.382 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:00:987.390 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:00:987.416 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:00:987.545 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244760987156 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 127110 calls = 19 average_bytes = 6690 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-03-23 Mon 05:46:00:987.559 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:02:876.323 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:02:876.462 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: SANDESH: Send FAILED: 1774244762876258 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-23 Mon 05:46:02:876.478 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-03-23 Mon 05:46:05:847.638 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:05:847.816 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:05:847.864 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:05:847.909 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:05:847.943 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:05:848.219 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:05:848.311 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:05:848.334 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:05:848.361 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:05:848.378 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:05:887.651 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:05:887.794 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:05:887.827 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:05:887.858 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:05:887.887 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:05:888.157 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:05:888.215 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:05:888.235 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:05:888.251 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:05:888.279 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:05:957.054 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:05:957.206 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:05:957.262 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:05:957.296 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:05:957.307 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:05:957.344 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:05:957.356 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:05:957.383 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:05:957.415 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:05:957.429 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:05:987.324 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:05:987.664 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:46:05:987.789 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:05:987.820 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244765987431 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:05:987.830 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:05:987.839 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:05:987.997 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244765987771 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 127110 calls = 19 average_bytes = 6690 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-03-23 Mon 05:46:05:988.024 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:05:991.378 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:51066::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:46:05:991.418 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:51066::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:46:05:991.429 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:51066::10.0.0.38:8086(-1):EvTcpConnected 2026-03-23 Mon 05:46:05:991.524 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:46:05:991.544 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:46:05:991.594 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: ClientInit 2026-03-23 Mon 05:46:05:991.618 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-3.:contrail-query-engine:0:Database count 4 2026-03-23 Mon 05:46:05:991.740 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:05:991.753 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:46:05:991.784 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: SANDESH: Send FAILED: 1774244765991585 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:05:991.803 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:46:05:991.815 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:05:991.825 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:46:05:991.840 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:05:991.850 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:46:05:992.113 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: SANDESH: Send FAILED: 1774244765991726 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 127110 calls = 19 average_bytes = 6690 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-03-23 Mon 05:46:05:992.144 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102206457600, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:46:07:621.203 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.33:51066::10.0.0.38:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2026-03-23 Mon 05:46:07:621.245 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.33:51066::10.0.0.38:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:07:621.268 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:51066::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-23 Mon 05:46:07:621.277 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:51066::10.0.0.38:8086(-1):EvTcpClose 2026-03-23 Mon 05:46:07:621.366 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-23 Mon 05:46:07:621.396 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: ClientInit : EvTcpClose 2026-03-23 Mon 05:46:07:621.522 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Return to idle with 5000 2026-03-23 Mon 05:46:07:621.532 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Idle 2026-03-23 Mon 05:46:07:621.607 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:07:621.754 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: SANDESH: Send FAILED: 1774244767621480 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:07:621.781 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:07:621.791 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:07:621.813 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: SANDESH: Send FAILED: 1774244767621509 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:07:621.821 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:07:621.829 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:07:621.855 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:07:621.987 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: SANDESH: Send FAILED: 1774244767621580 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:07:622.002 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102223251200, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:10:848.330 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:10:848.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:10:848.549 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:10:848.593 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:10:848.635 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:10:848.745 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:10:848.786 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:10:848.802 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:10:848.815 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:10:848.829 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:10:888.245 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:10:888.488 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:10:888.536 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:10:888.592 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:10:888.623 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:10:888.663 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:10:888.697 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:10:888.770 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:10:889.816 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:10:889.847 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:10:957.328 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:10:957.517 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:10:957.560 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:10:957.612 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:10:957.659 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:10:957.725 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:10:957.801 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:10:957.828 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:10:957.848 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:10:957.866 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:12:621.663 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:12:621.864 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:46:12:622.001 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:12:622.034 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244772621736 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:12:622.044 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:12:622.053 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:12:622.203 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244772621988 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:12:622.223 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:12:622.271 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:46:12:622.281 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:46:12:622.303 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:46:12:622.311 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:46:12:622.400 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:12:622.428 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Idle 2026-03-23 Mon 05:46:12:622.484 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:12:622.621 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244772622364 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:12:622.637 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:12:622.647 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:12:622.667 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244772622385 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:12:622.674 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:12:622.683 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:12:622.700 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:12:622.824 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244772622476 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:12:622.839 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:15:848.818 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:15:848.953 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:15:848.990 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:15:849.021 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:15:849.052 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:15:849.097 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:15:849.150 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:15:849.200 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:15:849.219 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:15:849.233 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:15:888.725 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:15:888.865 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:15:889.114 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:15:889.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:15:889.820 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:15:889.908 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:15:889.940 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:15:890.113 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:15:890.136 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:15:890.180 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:15:957.588 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:15:957.722 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:15:957.843 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:15:957.906 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:15:957.922 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:15:957.951 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:15:957.979 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:15:958.012 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:15:958.027 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:15:958.042 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:20:622.603 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:20:622.958 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-23 Mon 05:46:20:623.095 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:20:623.134 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: SANDESH: Send FAILED: 1774244780622717 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:20:623.148 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:20:623.176 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:20:623.349 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: SANDESH: Send FAILED: 1774244780623079 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:20:623.368 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:20:623.424 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:46:20:623.434 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:46:20:623.696 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:46:20:623.766 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:46:20:623.967 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:20:624.024 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Idle 2026-03-23 Mon 05:46:20:624.130 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:20:624.398 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: SANDESH: Send FAILED: 1774244780623888 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:20:624.499 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:20:624.542 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:20:624.579 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: SANDESH: Send FAILED: 1774244780623935 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:20:624.597 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:20:624.620 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:20:624.672 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:20:624.953 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: SANDESH: Send FAILED: 1774244780624097 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:20:625.007 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:20:849.203 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:20:849.395 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:20:849.446 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:20:849.510 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:20:849.560 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:20:849.629 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:20:849.687 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:20:849.708 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:20:849.745 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:20:849.765 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:20:889.209 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:20:889.420 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:20:889.798 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:20:889.846 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:20:890.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:20:890.213 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:20:890.260 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:20:890.448 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:20:890.490 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:20:890.510 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:20:957.811 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:20:958.056 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:20:958.085 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:20:958.119 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:20:958.158 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:20:958.173 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:20:958.184 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:20:958.211 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:20:958.241 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:20:958.255 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:25:849.714 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:25:849.882 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:25:849.934 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:25:850.026 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:25:850.102 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:25:850.150 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:25:850.179 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:25:850.193 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:25:850.242 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:25:850.263 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:25:889.926 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:25:890.085 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:25:890.289 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:25:890.320 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:25:890.505 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:25:890.545 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:25:890.576 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:25:890.715 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:25:890.749 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:25:890.762 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:25:958.120 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:25:958.297 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:25:958.324 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:25:958.357 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:25:958.384 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:25:958.412 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:25:958.444 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:25:958.458 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:25:958.471 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:25:958.483 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:27:881.820 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-23 Mon 05:46:27:881.986 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:27:882.082 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: SANDESH: Send FAILED: 1774244787881663 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 69 dequeues = 69 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 92 dequeues = 92 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244780624023 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244780624669 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:46:27:882.098 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:27:882.109 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:27:882.265 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: SANDESH: Send FAILED: 1774244787881797 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:27:882.296 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:30:850.227 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:30:850.395 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:30:850.427 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:30:850.459 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:30:850.771 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:30:850.842 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:30:850.889 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:30:850.916 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:30:850.929 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:30:850.943 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:30:890.365 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:30:890.512 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:30:890.818 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:30:890.855 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:30:890.866 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:30:890.929 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:30:890.974 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:30:891.133 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:30:891.183 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:30:891.265 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:30:958.380 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:30:958.597 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:30:958.630 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:30:958.669 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:30:958.698 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:30:958.726 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:30:958.769 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:30:958.790 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:30:958.807 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:30:958.824 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:35:850.922 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:35:851.071 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:35:851.113 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:35:851.147 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:35:851.186 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:35:851.403 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:35:851.437 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:35:851.453 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:35:851.466 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:35:851.480 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:35:890.886 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:35:891.061 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:35:891.163 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:35:891.292 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:35:891.324 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:35:891.384 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:35:891.419 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:35:891.434 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:35:891.560 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:35:891.628 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:35:958.665 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:35:958.826 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:35:958.850 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:35:958.882 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:35:958.930 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:35:958.946 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:35:958.956 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:35:958.982 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:35:959.012 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:35:959.026 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:36:624.140 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:36:624.373 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:46:36:624.530 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:36:624.569 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: SANDESH: Send FAILED: 1774244796624218 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:36:624.579 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:36:624.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:36:624.757 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: SANDESH: Send FAILED: 1774244796624505 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:36:624.782 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:36:624.837 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:46:36:624.846 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:46:36:624.885 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:46:36:624.916 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:46:36:625.034 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:36:625.074 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Idle 2026-03-23 Mon 05:46:36:625.143 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:36:625.285 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: SANDESH: Send FAILED: 1774244796624976 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:36:625.303 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:36:625.313 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:36:625.334 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: SANDESH: Send FAILED: 1774244796625004 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:36:625.342 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:36:625.350 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:36:625.360 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:36:625.483 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: SANDESH: Send FAILED: 1774244796625131 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:36:625.550 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102235846400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:40:851.494 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:40:851.649 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:40:851.685 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:40:851.719 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:40:851.776 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:40:851.821 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:40:851.862 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:40:851.876 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:40:851.890 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:40:851.921 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:40:891.363 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:40:891.523 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:40:891.567 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:40:891.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:40:891.664 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:40:891.794 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:40:891.850 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:40:891.873 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:40:891.913 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:40:891.934 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:40:958.888 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:40:959.068 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:40:959.100 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:40:959.141 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:40:959.188 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:40:959.228 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:40:959.277 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:40:959.312 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:40:959.334 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:40:959.462 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:45:851.917 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:45:852.070 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:45:852.104 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:45:852.135 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:45:852.164 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:45:852.234 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:45:852.283 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:45:852.321 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:45:852.336 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:45:852.348 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:45:891.900 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:45:892.093 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:45:892.138 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:45:892.169 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:45:892.195 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:45:892.322 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:45:892.354 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:45:892.376 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:45:892.389 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:45:959.150 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:45:959.369 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:45:959.399 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:45:959.431 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:45:959.480 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:45:959.496 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:45:959.506 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:45:959.542 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:45:959.553 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:45:959.583 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:45:960.933 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:50:852.306 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:50:852.454 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:50:852.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:50:852.520 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:50:852.576 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:50:852.627 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:50:852.651 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:50:852.671 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:50:852.684 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:50:852.696 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:50:892.471 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:50:892.588 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:50:892.620 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:50:892.649 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:50:892.768 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:50:892.797 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:50:892.827 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:50:892.843 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:50:959.446 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:50:959.610 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:50:959.637 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:50:959.670 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:50:959.698 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:50:959.725 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:50:959.756 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:50:959.770 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:50:959.795 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:50:959.807 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:50:960.978 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:50:961.391 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:55:852.704 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:46:55:852.852 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:46:55:852.916 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:46:55:852.960 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:46:55:852.996 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:46:55:853.047 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:46:55:853.082 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:46:55:853.102 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:46:55:853.116 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:46:55:853.131 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:46:55:892.832 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:46:55:892.998 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:46:55:893.041 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:46:55:893.069 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:46:55:893.278 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:46:55:893.427 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:46:55:893.460 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:46:55:893.483 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:46:55:959.675 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:46:55:959.849 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:46:55:959.877 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:46:55:959.942 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:46:55:959.987 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:46:55:960.025 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:46:55:960.060 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:46:55:960.077 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:46:55:960.090 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:46:55:960.102 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:46:55:961.441 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:46:55:961.940 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:46:57:882.152 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:57:882.257 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: SANDESH: Send FAILED: 1774244817882012 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 76 dequeues = 76 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 102 dequeues = 102 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244796625072 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244796625358 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:46:57:882.271 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:57:882.282 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:57:882.418 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: SANDESH: Send FAILED: 1774244817882109 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 55, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:57:882.441 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102227449600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:00:853.114 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:47:00:853.234 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:47:00:853.267 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:47:00:853.313 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:47:00:853.351 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:47:00:853.402 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:47:00:853.440 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:47:00:853.460 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:47:00:853.474 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:47:00:853.486 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:47:00:893.389 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:47:00:893.510 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:47:00:893.545 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:47:00:893.572 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:47:00:893.659 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:47:00:893.711 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:47:00:893.733 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:47:00:893.803 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:47:00:959.928 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:47:00:960.100 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:47:00:960.127 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:47:00:960.167 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:47:00:960.180 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:47:00:960.206 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:47:00:960.233 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:47:00:960.266 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:47:00:960.281 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:47:00:960.293 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:47:00:962.021 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:47:00:962.335 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:47:02:878.095 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:02:878.195 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: SANDESH: Send FAILED: 1774244822878062 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-23 Mon 05:47:02:878.207 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102202259200, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-03-23 Mon 05:47:05:853.502 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:47:05:853.643 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:47:05:853.678 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:47:05:854.035 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:47:05:854.077 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:47:05:854.090 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:47:05:854.123 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:47:05:854.154 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:47:05:854.177 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:47:05:854.230 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:47:05:894.967 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:47:05:895.076 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:47:05:895.112 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:47:05:895.141 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:47:05:895.327 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:47:05:895.389 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:47:05:895.414 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:47:05:895.437 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:47:05:960.192 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:47:05:960.373 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:47:05:960.414 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:47:05:960.468 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:47:05:960.504 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:47:05:960.532 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:47:05:960.564 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:47:05:960.580 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:47:05:960.592 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:47:05:960.604 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:47:05:962.382 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:47:05:962.736 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:47:06:625.700 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:47:06:626.359 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:47:06:626.472 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:06:626.502 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SANDESH: Send FAILED: 1774244826626210 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:47:06:626.511 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:06:626.520 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:06:626.662 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SANDESH: Send FAILED: 1774244826626456 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 58, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:06:626.687 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:06:626.718 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:47:06:626.727 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:47:06:626.761 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:47:06:626.770 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:47:06:627.287 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:47:06:627.320 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Idle 2026-03-23 Mon 05:47:06:627.382 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:06:627.524 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SANDESH: Send FAILED: 1774244826626827 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:06:627.543 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:06:627.552 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:06:627.573 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SANDESH: Send FAILED: 1774244826626851 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:47:06:627.580 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:06:627.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:47:06:627.598 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:06:627.719 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SANDESH: Send FAILED: 1774244826627368 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:06:627.735 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:10:854.117 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:47:10:854.290 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:47:10:854.339 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:47:10:854.381 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:47:10:854.420 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:47:10:854.479 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f86a0 2026-03-23 Mon 05:47:10:854.510 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7d60 2026-03-23 Mon 05:47:10:854.531 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f7420 2026-03-23 Mon 05:47:10:854.551 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f6ae0 2026-03-23 Mon 05:47:10:854.570 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f61a0 2026-03-23 Mon 05:47:10:895.430 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:47:10:895.582 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:47:10:895.631 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:47:10:895.677 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:47:10:895.800 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5870 2026-03-23 Mon 05:47:10:895.849 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f5340 2026-03-23 Mon 05:47:10:895.865 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f4ca0 2026-03-23 Mon 05:47:10:895.879 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28ee1d0 2026-03-23 Mon 05:47:10:960.469 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:47:10:960.628 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fb4e0 2026-03-23 Mon 05:47:10:960.653 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:47:10:960.691 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:47:10:960.732 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:47:10:960.759 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:47:10:960.791 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28faba0 2026-03-23 Mon 05:47:10:960.807 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28fa260 2026-03-23 Mon 05:47:10:960.819 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f9920 2026-03-23 Mon 05:47:10:960.831 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f8fe0 2026-03-23 Mon 05:47:10:962.807 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:47:10:963.112 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: -1 0x28f45f0 2026-03-23 Mon 05:47:15:854.659 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f86a0 2026-03-23 Mon 05:47:15:854.863 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7d60 2026-03-23 Mon 05:47:15:854.950 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f7420 2026-03-23 Mon 05:47:15:855.003 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f6ae0 2026-03-23 Mon 05:47:15:855.046 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f61a0 2026-03-23 Mon 05:47:15:855.097 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f86a0 2026-03-23 Mon 05:47:15:855.107 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:855.151 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:855.185 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f7d60 2026-03-23 Mon 05:47:15:855.198 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:855.212 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:855.230 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f7420 2026-03-23 Mon 05:47:15:855.240 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:855.254 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:855.266 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f6ae0 2026-03-23 Mon 05:47:15:855.274 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:855.287 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:855.357 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f61a0 2026-03-23 Mon 05:47:15:855.367 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:855.381 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:855.542 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:855.562 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:855.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:855.604 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:855.645 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:856.857 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:15:856.931 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244835855691 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:15:856.946 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:15:856.986 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:857.021 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:857.038 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:857.060 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:857.085 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:857.118 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:857.146 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:857.168 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:857.184 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:857.215 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:895.926 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5870 2026-03-23 Mon 05:47:15:896.102 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f5340 2026-03-23 Mon 05:47:15:896.178 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f4ca0 2026-03-23 Mon 05:47:15:896.235 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28ee1d0 2026-03-23 Mon 05:47:15:896.316 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f5870 2026-03-23 Mon 05:47:15:896.332 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:896.362 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:896.398 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f5340 2026-03-23 Mon 05:47:15:896.408 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:896.454 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:896.470 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f4ca0 2026-03-23 Mon 05:47:15:896.479 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:896.492 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:896.543 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28ee1d0 2026-03-23 Mon 05:47:15:896.552 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:896.567 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:899.283 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:899.323 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:899.340 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:899.350 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:899.363 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:899.372 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:899.385 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:899.394 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:899.423 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:899.493 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:899.519 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:899.541 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:15:899.614 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SANDESH: Send FAILED: 1774244835899454 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:15:899.628 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:15:899.543 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:960.685 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fb4e0 2026-03-23 Mon 05:47:15:960.890 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28fb4e0 2026-03-23 Mon 05:47:15:960.917 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:960.939 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:960.948 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28faba0 2026-03-23 Mon 05:47:15:960.999 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28fa260 2026-03-23 Mon 05:47:15:961.040 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f9920 2026-03-23 Mon 05:47:15:961.079 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f8fe0 2026-03-23 Mon 05:47:15:961.151 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28faba0 2026-03-23 Mon 05:47:15:961.159 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:961.170 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:961.179 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28fa260 2026-03-23 Mon 05:47:15:961.185 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:961.193 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:961.201 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f9920 2026-03-23 Mon 05:47:15:961.206 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:961.213 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:961.221 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f8fe0 2026-03-23 Mon 05:47:15:961.226 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:961.234 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:961.266 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:961.276 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:961.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:961.512 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:961.522 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:962.064 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:15:962.099 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244835961551 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:15:962.108 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:15:962.128 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:962.147 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:962.157 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:962.170 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:962.179 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:962.192 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:962.201 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:962.218 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:962.232 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:962.252 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:963.150 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x28f45f0 2026-03-23 Mon 05:47:15:963.512 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: RAC_Connect status: 0 0x28f45f0 2026-03-23 Mon 05:47:15:963.544 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:963.565 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:964.113 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:964.144 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:964.177 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:27:882.339 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-23 Mon 05:47:27:882.390 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:27:882.453 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244847882224 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 87 dequeues = 87 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 116 dequeues = 116 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244826627319 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244835962060 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:47:27:882.466 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:27:882.475 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:27:882.608 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: SANDESH: Send FAILED: 1774244847882323 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:27:882.632 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101094995712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:36:628.409 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101090797312, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:47:36:629.523 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101090797312, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-23 Mon 05:47:36:629.638 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101090797312, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:36:629.668 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101090797312, Pid 1]: SANDESH: Send FAILED: 1774244856629351 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:36:629.677 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101090797312, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:36:629.688 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101090797312, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:36:629.841 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101090797312, Pid 1]: SANDESH: Send FAILED: 1774244856629621 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:36:629.856 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140101090797312, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:36:634.251 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:50432::10.0.0.33:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:47:36:634.278 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:50432::10.0.0.33:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:47:36:634.287 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102470125696, Pid 1]: 10.0.0.33:50432::10.0.0.33:8086(-1):EvTcpConnected 2026-03-23 Mon 05:47:36:637.551 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:47:36:637.583 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:47:36:637.624 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: ClientInit 2026-03-23 Mon 05:47:36:637.644 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-3.:contrail-query-engine:0:Database count 5 2026-03-23 Mon 05:47:36:637.840 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:47:36:637.853 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:47:36:637.878 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SANDESH: Send FAILED: 1774244856637616 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:36:637.886 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:47:36:637.908 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:47:36:637.916 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:47:36:637.928 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:47:36:637.935 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:47:36:638.090 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SANDESH: Send FAILED: 1774244856637828 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 70, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:36:638.115 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:47:36:650.076 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:47:36:650.142 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:47:36:650.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:47:36:650.178 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:47:36:650.454 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 128683 calls = 20 average_bytes = 6434 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 87 dequeues = 87 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 116 dequeues = 116 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244826627319 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244835962060 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 14 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 96674 calls = 13 average_bytes = 7436 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 70, _iter59->first = sent _iter59->second = 21, ] ] max_sm_queue_count = 5 proxy seq 49 2026-03-23 Mon 05:47:36:650.500 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:47:36:650.527 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 35 2026-03-23 Mon 05:47:36:650.541 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:47:36:650.573 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 4 2026-03-23 Mon 05:47:36:650.584 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:47:36:650.592 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:47:36:650.625 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: Established 2026-03-23 Mon 05:47:42:669.295 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: OnMessage control in state: Established 2026-03-23 Mon 05:47:42:669.394 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-03-23 Mon 05:47:42:669.413 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Established : EvSandeshMessageRecv 2026-03-23 Mon 05:47:42:669.456 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Received Ctrl Message with size 3 2026-03-23 Mon 05:47:42:670.253 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 50, total = 0 2026-03-23 Mon 05:47:42:670.267 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 37, total = 0 2026-03-23 Mon 05:47:42:670.276 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 4, total = 0 2026-03-23 Mon 05:47:42:670.284 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:48:27:884.516 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102322255616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-23 Mon 05:49:27:886.219 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102231648000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-23 Mon 05:50:27:888.010 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102231648000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-23 Mon 05:51:27:889.899 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-23 Mon 05:52:27:890.463 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102313858816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-03-23 Mon 05:53:08:510.918 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvStop in state Established 2026-03-23 Mon 05:53:08:511.030 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Idle 2026-03-23 Mon 05:53:08:511.128 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:53:08:511.140 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:53:08:511.290 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: SANDESH: Send FAILED: 1774245188511116 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774244607794718 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3463 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 448390 calls = 52 average_bytes = 8622 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 32 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 72, _iter59->first = sent _iter59->second = 56, ] ] ] 2026-03-23 Mon 05:53:08:511.315 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140102326454016, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:53:33:809.094 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: http-server-port 8091 2026-03-23 Mon 05:53:33:809.169 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Max-tasks 16 2026-03-23 Mon 05:53:33:809.176 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Max-slice 100 2026-03-23 Mon 05:53:33:809.182 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Collectors 10.0.0.38:8086 2026-03-23 Mon 05:53:33:809.187 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Collectors 10.0.0.50:8086 2026-03-23 Mon 05:53:33:809.192 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Collectors 10.0.0.33:8086 2026-03-23 Mon 05:53:33:809.211 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: ROLE : Generator 2026-03-23 Mon 05:53:33:809.216 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-23 Mon 05:53:33:809.221 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5515-3. 2026-03-23 Mon 05:53:33:809.226 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-23 Mon 05:53:33:814.616 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-23 Mon 05:53:33:814.625 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-23 Mon 05:53:33:814.631 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-23 Mon 05:53:33:819.508 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: HTTP Introspect Init 2026-03-23 Mon 05:53:33:819.626 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 138 2026-03-23 Mon 05:53:33:819.715 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Sandesh Http Server Port: 8091 2026-03-23 Mon 05:53:33:819.772 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-23 Mon 05:53:33:819.788 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-23 Mon 05:53:33:820.124 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Idle 2026-03-23 Mon 05:53:33:820.255 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: No Client: 1774245213820202 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245213820135 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:53:33:820.391 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-23 Mon 05:53:33:820.405 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-23 Mon 05:53:33:820.416 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-03-23 Mon 05:53:33:820.516 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Processing scm::EvStart in state Idle 2026-03-23 Mon 05:53:33:820.597 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Disconnect 2026-03-23 Mon 05:53:33:820.663 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-23 Mon 05:53:33:820.846 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:53:33:820.901 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:53:33:820.928 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: SANDESH: Send FAILED: 1774245213820591 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:53:33:820.943 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:53:33:820.951 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:53:33:821.014 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: SANDESH: Send FAILED: 1774245213820652 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245213820135 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:53:33:821.025 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:53:33:821.033 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:53:33:821.077 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: SANDESH: Send FAILED: 1774245213820693 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245213820135 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:53:33:821.088 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:53:33:821.096 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:53:33:821.107 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: SANDESH: Send FAILED: 1774245213820709 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:53:33:821.113 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:53:33:821.120 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:53:33:821.160 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: SANDESH: Send FAILED: 1774245213820890 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245213820135 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:53:33:821.169 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:53:33:925.836 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-23 Mon 05:53:33:936.490 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:53:33:936.539 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: SANDESH: Send FAILED: 1774245213935967 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:53:33:936.554 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:53:33:938.021 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:47596::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2026-03-23 Mon 05:53:33:938.037 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: 10.0.0.33:47596::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:53:33:938.044 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: 10.0.0.33:47596::10.0.0.50:8086(-1):EvTcpConnected 2026-03-23 Mon 05:53:33:938.086 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20da0f0 2026-03-23 Mon 05:53:33:938.093 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.102 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.112 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:53:33:938.114 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20dad30 2026-03-23 Mon 05:53:33:938.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.166 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.186 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20d6430 2026-03-23 Mon 05:53:33:938.191 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.200 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.210 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20db970 2026-03-23 Mon 05:53:33:938.215 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.222 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.232 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20ce540 2026-03-23 Mon 05:53:33:938.237 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.243 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.253 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20dc5b0 2026-03-23 Mon 05:53:33:938.263 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.270 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.279 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20d3540 2026-03-23 Mon 05:53:33:938.284 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.291 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.300 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20d6ff0 2026-03-23 Mon 05:53:33:938.305 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.311 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.320 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20d7c30 2026-03-23 Mon 05:53:33:938.325 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.331 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.340 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20d4ab0 2026-03-23 Mon 05:53:33:938.345 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.352 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.361 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20d8870 2026-03-23 Mon 05:53:33:938.366 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.372 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.381 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20d3df0 2026-03-23 Mon 05:53:33:938.386 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.392 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.401 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20dd1f0 2026-03-23 Mon 05:53:33:938.406 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.412 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.425 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20d94b0 2026-03-23 Mon 05:53:33:938.430 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.436 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.445 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: RAC_Connect status: 0 0x20d5770 2026-03-23 Mon 05:53:33:938.450 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:53:33:938.456 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:53:33:938.646 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:53:33:938.683 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: ClientInit 2026-03-23 Mon 05:53:33:938.712 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-3.:contrail-query-engine:0:Database count 1 2026-03-23 Mon 05:53:33:938.788 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:53:33:938.797 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:53:33:938.813 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: SANDESH: Send FAILED: 1774245213938678 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:53:33:938.820 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:53:33:938.827 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:53:33:938.833 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:53:33:938.852 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:53:33:938.858 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:53:33:938.951 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: SANDESH: Send FAILED: 1774245213938777 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774245213820135 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:53:33:938.971 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:53:33:940.563 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.592 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.606 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.625 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.643 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.651 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.662 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.670 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.680 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.688 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.699 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.706 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.717 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.724 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.735 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.742 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.759 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.766 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.777 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.784 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.795 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.802 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.814 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.821 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.832 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.839 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.849 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.857 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.867 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:53:33:940.874 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:53:33:940.936 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:940.946 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:53:33:940.976 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:53:33:940.960 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.013 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.018 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: SANDESH: Send FAILED: 1774245213940918 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:53:33:941.025 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.053 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.072 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.092 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.103 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.113 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.124 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.137 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.152 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.162 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.171 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.180 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:53:33:941.034 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:53:33:941.250 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:53:33:941.257 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:53:33:941.274 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: SANDESH: Send FAILED: 1774245213940956 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:53:33:941.281 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:53:33:941.288 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:53:33:941.294 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:53:33:941.305 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: SANDESH: Send FAILED: 1774245213941087 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:53:33:941.311 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:53:33:941.318 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:53:33:941.323 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:53:33:941.334 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: SANDESH: Send FAILED: 1774245213941149 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:53:33:941.344 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533444974144, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:53:33:953.775 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:53:33:954.596 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:53:33:954.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:53:33:954.644 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:53:33:954.739 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774245213820135 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2026-03-23 Mon 05:53:33:954.766 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:53:33:954.788 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2026-03-23 Mon 05:53:33:954.801 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:53:33:954.808 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:53:33:954.818 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:53:33:954.860 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: Established 2026-03-23 Mon 05:54:33:820.974 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533192177216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-23 Mon 05:55:33:821.497 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-23 Mon 05:56:33:822.136 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533208970816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-23 Mon 05:57:33:822.648 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533453370944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-23 Mon 05:58:33:823.291 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533200574016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-23 Mon 05:59:33:823.841 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533449172544, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-23 Mon 06:00:33:824.438 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533453370944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-23 Mon 06:01:33:826.000 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-23 Mon 06:02:33:826.708 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533208970816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-03-23 Mon 06:03:33:827.302 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2026-03-23 Mon 06:04:33:827.779 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533440775744, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2026-03-23 Mon 06:05:33:828.363 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533449172544, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2026-03-23 Mon 06:06:33:828.972 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533192177216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2026-03-23 Mon 06:07:33:829.551 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533196375616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2026-03-23 Mon 06:07:41:321.824 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.33:54648 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:08:00:013.437 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.33:38888 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:08:33:830.771 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2026-03-23 Mon 06:09:28:945.180 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.38:57604 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:09:33:831.439 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2026-03-23 Mon 06:09:46:773.706 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.38 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-03-23 Mon 06:09:46:774.362 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID e49222c7-267e-11f1-997e-00000a000026 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-03-23 Mon 06:09:46:774.428 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QEQueryLog: e49222c7-267e-11f1-997e-00000a000026 :0:json query is: { end_time: 1774246186761605 , enqueue_time: 1774246186773481 , query_metadata: {"enqueue_time": 1774246186773370} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333:ctest-vn22-62662333", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:46:774.690 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: Chunks: 2 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-03-23 Mon 06:09:46:774.712 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-03-23 Mon 06:09:46:774.759 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for e49222c7-267e-11f1-997e-00000a000026 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-03-23 Mon 06:09:46:774.916 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533453370944, Pid 1]: [SYS_INFO]: QEQueryLog: e49222c7-267e-11f1-997e-00000a000026 :0:json query is: { end_time: 1774246186761605 , enqueue_time: 1774246186773481 , query_metadata: {"enqueue_time": 1774246186773370} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333:ctest-vn22-62662333", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:46:825.233 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:46:825.294 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:46:825.304 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:46:825.311 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:46:825.318 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 UNION between tables of sizes 0 and 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:46:825.345 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:46:825.352 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:46:825.359 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:46:825.366 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:46:825.372 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:46:825.411 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:46:825.418 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:46:825.424 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:46:825.431 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:46:825.437 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:46:825.443 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:46:825.450 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:46:825.456 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:46:825.462 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:46:825.468 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: e49222c7-267e-11f1-997e-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:46:825.537 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QEQueryLog: e49222c7-267e-11f1-997e-00000a000026 :0:json query is: { end_time: 1774246186761605 , enqueue_time: 1774246186773481 , query_metadata: {"enqueue_time": 1774246186773370} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333:ctest-vn22-62662333", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:46:825.969 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QEQueryLog: e49222c7-267e-11f1-997e-00000a000026 ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:46:826.154 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-03-23 Mon 06:09:46:826.181 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-03-23 Mon 06:09:46:826.227 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-03-23 Mon 06:09:46:827.191 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533187978816, Pid 1]: [SYS_INFO]: QELog: Finished: QID e49222c7-267e-11f1-997e-00000a000026 Table ObjectRoutingInstance Time(ms) 52 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-03-23 Mon 06:09:46:827.204 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533428180544, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5515-3. table = ObjectRoutingInstance query_stats= [ qid = e49222c7-267e-11f1-997e-00000a000026 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333:ctest-vn22-62662333", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 2 chunk_where_time = 50,0, 50,0, chunk_select_time = 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 52 rows = 1 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-03-23 Mon 06:09:46:827.420 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533187978816, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-03-23 Mon 06:10:33:832.250 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2026-03-23 Mon 06:11:33:832.785 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533187978816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2026-03-23 Mon 06:12:33:833.327 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533162796608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2026-03-23 Mon 06:13:33:833.983 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533187978816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2026-03-23 Mon 06:14:33:835.389 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533187978816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2026-03-23 Mon 06:15:33:835.921 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533187978816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2026-03-23 Mon 06:16:33:836.714 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533196375616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2026-03-23 Mon 06:17:33:839.033 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533449172544, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2026-03-23 Mon 06:18:33:839.829 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533449172544, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2026-03-23 Mon 06:19:33:840.461 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533204772416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2026-03-23 Mon 06:20:33:840.995 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533457569344, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2026-03-23 Mon 06:21:08:099.050 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.38:45702 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:21:33:842.091 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533440775744, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2026-03-23 Mon 06:22:33:842.705 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533432378944, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2026-03-23 Mon 06:22:56:522.248 UTC cn-jenkins-deploy-platform-ansible-os-5515-3 [Thread 140533464709248, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.33:56456 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256