2026-03-23 Mon 05:37:10:785.475 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: http-server-port 8091 2026-03-23 Mon 05:37:10:785.566 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Max-tasks 16 2026-03-23 Mon 05:37:10:785.575 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Max-slice 100 2026-03-23 Mon 05:37:10:785.581 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Collectors 10.0.0.38:8086 2026-03-23 Mon 05:37:10:785.586 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Collectors 10.0.0.50:8086 2026-03-23 Mon 05:37:10:785.591 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Collectors 10.0.0.33:8086 2026-03-23 Mon 05:37:10:785.610 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: ROLE : Generator 2026-03-23 Mon 05:37:10:785.616 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-23 Mon 05:37:10:785.621 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5515-2. 2026-03-23 Mon 05:37:10:785.626 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-23 Mon 05:37:10:785.632 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-23 Mon 05:37:10:785.637 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-23 Mon 05:37:10:785.643 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-23 Mon 05:37:10:786.196 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: HTTP Introspect Init 2026-03-23 Mon 05:37:10:786.823 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-03-23 Mon 05:37:10:786.896 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Sandesh Http Server Port: 8091 2026-03-23 Mon 05:37:10:786.956 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-23 Mon 05:37:10:786.966 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-23 Mon 05:37:10:787.046 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Idle 2026-03-23 Mon 05:37:10:787.198 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: No Client: 1774244230787126 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230787055 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:10:787.321 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-23 Mon 05:37:10:787.336 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-23 Mon 05:37:10:787.349 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-03-23 Mon 05:37:10:787.625 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Processing scm::EvStart in state Idle 2026-03-23 Mon 05:37:10:787.692 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Disconnect 2026-03-23 Mon 05:37:10:787.760 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-23 Mon 05:37:10:787.953 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:37:10:788.021 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:788.042 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: SANDESH: Send FAILED: 1774244230787684 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:10:788.052 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:10:788.060 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:801.526 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: SANDESH: Send FAILED: 1774244230787750 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230787055 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:10:801.579 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:10:801.599 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:801.787 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: SANDESH: Send FAILED: 1774244230787808 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230787055 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:10:801.804 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:10:801.815 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:801.833 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: SANDESH: Send FAILED: 1774244230787827 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:10:801.842 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:10:801.854 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:801.944 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: SANDESH: Send FAILED: 1774244230788010 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230787055 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:10:801.959 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:10:802.042 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:10:802.054 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:10:802.107 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:10:802.140 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:802.161 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: SANDESH: Send FAILED: 1774244230802087 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:10:802.168 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:15:805.907 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:15:805.969 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:15:806.006 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:20:807.566 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:20:807.737 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:20:807.762 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:25:809.476 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:25:809.522 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:25:809.543 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:31:232.389 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-23 Mon 05:37:31:232.438 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:31:232.459 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:36:281.216 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-23 Mon 05:37:36:281.263 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:36:281.284 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:41:366.024 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:41:366.100 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:46:391.509 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-23 Mon 05:37:51:395.380 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388723586816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:51:395.465 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388723586816, Pid 1]: SANDESH: Send FAILED: 1774244271395294 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:51:395.477 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388723586816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:395.418 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:37:51:395.945 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:37:51:395.977 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xca85f0 2026-03-23 Mon 05:37:51:395.984 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:395.999 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.012 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xca21d0 2026-03-23 Mon 05:37:51:396.018 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.028 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.038 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xca8d10 2026-03-23 Mon 05:37:51:396.044 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.051 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:37:51:396.123 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:37:51:396.054 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.177 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xca93b0 2026-03-23 Mon 05:37:51:396.184 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.235 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.283 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xca98e0 2026-03-23 Mon 05:37:51:396.297 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:37:51:396.297 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.393 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.401 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Idle 2026-03-23 Mon 05:37:51:396.412 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xcaa210 2026-03-23 Mon 05:37:51:396.426 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.448 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.467 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xcaab50 2026-03-23 Mon 05:37:51:396.477 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.479 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:396.502 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.548 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xcab490 2026-03-23 Mon 05:37:51:396.557 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.571 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.586 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xcabdd0 2026-03-23 Mon 05:37:51:396.594 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.608 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.622 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xcac710 2026-03-23 Mon 05:37:51:396.630 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.643 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: SANDESH: Send FAILED: 1774244271396226 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230787055 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 24 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 25, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:51:396.645 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.668 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:396.691 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xcad050 2026-03-23 Mon 05:37:51:396.703 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.705 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:396.723 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.741 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xcad990 2026-03-23 Mon 05:37:51:396.747 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.757 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.770 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xcae2d0 2026-03-23 Mon 05:37:51:396.778 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.786 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.796 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xcaec10 2026-03-23 Mon 05:37:51:396.801 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.808 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.817 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: 0 0xcaf550 2026-03-23 Mon 05:37:51:396.822 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.830 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.747 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: SANDESH: Send FAILED: 1774244271396264 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:51:396.912 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:396.939 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:37:51:396.983 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:397.094 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: SANDESH: Send FAILED: 1774244271396466 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230787055 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 27 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 29, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:51:397.111 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:397.218 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:397.275 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: SANDESH: Send FAILED: 1774244271397142 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244271396399 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244271396981 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:37:51:397.286 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:397.299 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:397.451 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: SANDESH: Send FAILED: 1774244271397204 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230787055 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 36 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 38, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:51:397.475 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:397.596 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:397.614 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:397.629 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:397.639 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:397.651 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:397.660 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:397.677 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:397.686 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:397.698 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:397.707 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:397.763 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:397.794 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:397.807 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388505466624, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:397.814 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:397.858 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:397.868 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388505466624, Pid 1]: SANDESH: Send FAILED: 1774244271397743 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:37:51:397.876 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:397.909 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:397.919 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:397.939 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:397.949 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:397.966 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:397.975 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:397.987 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:397.996 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.008 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.017 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.051 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.078 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.094 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.108 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.123 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:397.886 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388505466624, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:398.343 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388505466624, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:398.367 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388505466624, Pid 1]: SANDESH: Send FAILED: 1774244271397787 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.50:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:51:398.375 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388505466624, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:398.392 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388505466624, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:398.411 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388505466624, Pid 1]: SANDESH: Send FAILED: 1774244271398071 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:51:398.418 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388505466624, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:398.983 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.999 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:399.014 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:399.047 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:399.070 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:399.112 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:399.127 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:399.167 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: SANDESH: Send FAILED: 1774244271399093 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:51:399.176 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388806481664, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:399.130 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:399.387 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:399.408 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:399.422 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:399.439 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:399.448 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:399.464 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:399.478 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:399.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:52:396.524 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:37:52:396.788 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:37:52:396.899 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:52:396.939 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: SANDESH: Send FAILED: 1774244272396603 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:52:396.949 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:52:396.958 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:52:397.067 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: SANDESH: Send FAILED: 1774244272396885 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230787055 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:52:397.089 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:52:400.883 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:50188::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:37:52:400.915 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: 10.0.0.50:50188::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:37:52:400.936 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: 10.0.0.50:50188::10.0.0.50:8086(-1):EvTcpConnected 2026-03-23 Mon 05:37:52:401.002 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:37:52:401.028 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:37:52:401.064 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: ClientInit 2026-03-23 Mon 05:37:52:401.081 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-2.:contrail-query-engine:0:Database count 1 2026-03-23 Mon 05:37:52:401.161 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:37:52:401.170 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:37:52:401.193 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: SANDESH: Send FAILED: 1774244272401056 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:52:401.201 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:37:52:401.210 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:37:52:401.217 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:37:52:401.227 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:37:52:401.234 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:37:52:401.360 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: SANDESH: Send FAILED: 1774244272401150 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244230787055 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:52:401.409 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:37:52:409.244 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:37:52:409.298 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:37:52:409.313 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:37:52:409.347 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:37:52:409.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244230787055 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244271396399 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244271396981 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2026-03-23 Mon 05:37:52:409.632 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:37:52:409.670 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2026-03-23 Mon 05:37:52:409.689 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:37:52:409.706 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:37:52:409.718 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:37:52:409.760 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388719388416, Pid 1]: Established 2026-03-23 Mon 05:38:00:037.682 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388522260224, Pid 1]: OnMessage control in state: Established 2026-03-23 Mon 05:38:00:037.746 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388522260224, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-03-23 Mon 05:38:00:037.757 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388522260224, Pid 1]: Established : EvSandeshMessageRecv 2026-03-23 Mon 05:38:00:037.798 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388522260224, Pid 1]: Received Ctrl Message with size 3 2026-03-23 Mon 05:38:00:037.813 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388522260224, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2026-03-23 Mon 05:38:00:037.823 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388522260224, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2026-03-23 Mon 05:38:00:037.835 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388522260224, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2026-03-23 Mon 05:38:00:037.843 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388522260224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:38:21:397.536 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388522260224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-23 Mon 05:39:21:398.026 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388522260224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-23 Mon 05:40:21:398.588 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388677416704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-23 Mon 05:41:08:591.314 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:35286 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-23 Mon 05:41:21:399.537 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388685813504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-23 Mon 05:41:21:486.248 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:49328 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-23 Mon 05:42:21:401.093 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388677416704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-23 Mon 05:43:14:841.344 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Disconnect status: -1 0xcac710 2026-03-23 Mon 05:43:14:841.401 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.534 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Disconnect status: -1 0xcabdd0 2026-03-23 Mon 05:43:14:841.550 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.583 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Disconnect status: -1 0xcab490 2026-03-23 Mon 05:43:14:841.594 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Disconnect status: -1 0xcaab50 2026-03-23 Mon 05:43:14:841.627 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.652 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:43:14:841.659 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Disconnect status: -1 0xcaa210 2026-03-23 Mon 05:43:14:841.667 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:843.051 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: -1 0xcac710 2026-03-23 Mon 05:43:14:843.110 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: -1 0xcabdd0 2026-03-23 Mon 05:43:14:843.126 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: -1 0xcab490 2026-03-23 Mon 05:43:14:843.139 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: -1 0xcaab50 2026-03-23 Mon 05:43:14:843.151 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388954351744, Pid 1]: RAC_Connect status: -1 0xcaa210 2026-03-23 Mon 05:43:14:858.780 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388677416704, Pid 1]: Processing scm::EvStop in state Established 2026-03-23 Mon 05:43:14:858.849 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388677416704, Pid 1]: Idle 2026-03-23 Mon 05:43:14:858.939 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388677416704, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:43:14:858.975 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388677416704, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:43:14:859.117 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388677416704, Pid 1]: SANDESH: Send FAILED: 1774244594858927 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244230787055 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 286229 calls = 32 average_bytes = 8944 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 22 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 33, ] ] ] 2026-03-23 Mon 05:43:14:859.138 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140388677416704, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:31:693.794 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: http-server-port 8091 2026-03-23 Mon 05:43:31:693.873 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Max-tasks 16 2026-03-23 Mon 05:43:31:693.881 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Max-slice 100 2026-03-23 Mon 05:43:31:693.887 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Collectors 10.0.0.38:8086 2026-03-23 Mon 05:43:31:693.893 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Collectors 10.0.0.50:8086 2026-03-23 Mon 05:43:31:693.898 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Collectors 10.0.0.33:8086 2026-03-23 Mon 05:43:31:693.916 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: ROLE : Generator 2026-03-23 Mon 05:43:31:693.922 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-23 Mon 05:43:31:693.928 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5515-2. 2026-03-23 Mon 05:43:31:693.933 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-23 Mon 05:43:31:693.939 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-23 Mon 05:43:31:693.944 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-23 Mon 05:43:31:693.959 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-23 Mon 05:43:31:714.597 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: HTTP Introspect Init 2026-03-23 Mon 05:43:31:714.732 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-03-23 Mon 05:43:31:714.792 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Sandesh Http Server Port: 8091 2026-03-23 Mon 05:43:31:714.823 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-23 Mon 05:43:31:714.838 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-23 Mon 05:43:31:714.911 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Idle 2026-03-23 Mon 05:43:31:715.079 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: No Client: 1774244611715004 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:43:31:715.198 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-23 Mon 05:43:31:715.212 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-23 Mon 05:43:31:715.225 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-03-23 Mon 05:43:31:715.325 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvStart in state Idle 2026-03-23 Mon 05:43:31:715.394 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Disconnect 2026-03-23 Mon 05:43:31:715.462 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-23 Mon 05:43:31:715.731 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-23 Mon 05:43:31:715.831 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:31:715.853 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244611715385 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:43:31:715.868 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:43:31:715.878 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:31:715.969 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244611715452 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:43:31:715.982 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:31:715.991 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:31:716.063 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244611715509 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:43:31:716.073 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:31:716.082 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:31:716.097 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244611715529 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:43:31:716.103 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:43:31:716.114 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:31:716.175 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244611715815 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:43:31:716.184 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:32:075.250 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-23 Mon 05:43:57:102.093 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:57:102.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: SANDESH: Send FAILED: 1774244637102005 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:43:57:102.170 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:43:57:104.487 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:38770::10.0.0.33:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:43:57:104.513 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:38770::10.0.0.33:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:43:57:104.521 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:38770::10.0.0.33:8086(-1):EvTcpConnected 2026-03-23 Mon 05:43:57:104.562 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe025f0 2026-03-23 Mon 05:43:57:104.569 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:104.582 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:104.591 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:43:57:104.625 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:43:57:104.666 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: ClientInit 2026-03-23 Mon 05:43:57:104.683 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-2.:contrail-query-engine:0:Database count 1 2026-03-23 Mon 05:43:57:104.778 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:104.789 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:43:57:104.595 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe02ca0 2026-03-23 Mon 05:43:57:105.249 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:104.810 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: SANDESH: Send FAILED: 1774244637104658 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:43:57:105.268 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:105.288 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:43:57:105.310 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xdfc1d0 2026-03-23 Mon 05:43:57:105.317 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:105.317 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:105.328 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:105.339 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:43:57:105.350 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe03340 2026-03-23 Mon 05:43:57:105.365 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:105.365 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:105.377 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:43:57:105.388 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:105.441 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe03870 2026-03-23 Mon 05:43:57:105.446 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:105.458 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:105.468 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe041a0 2026-03-23 Mon 05:43:57:105.473 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:105.485 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:105.495 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe04ae0 2026-03-23 Mon 05:43:57:105.500 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:105.509 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:105.509 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: SANDESH: Send FAILED: 1774244637104767 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 4, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:43:57:105.520 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe05420 2026-03-23 Mon 05:43:57:105.555 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:105.546 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:43:57:105.565 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:108.677 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe05d60 2026-03-23 Mon 05:43:57:108.694 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:108.709 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:108.719 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe066a0 2026-03-23 Mon 05:43:57:108.724 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:108.733 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:108.743 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe06fe0 2026-03-23 Mon 05:43:57:108.748 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:108.756 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:108.766 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe07920 2026-03-23 Mon 05:43:57:108.771 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:108.779 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:108.788 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe08260 2026-03-23 Mon 05:43:57:108.793 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:108.800 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:108.810 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe08ba0 2026-03-23 Mon 05:43:57:108.815 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:108.822 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:108.831 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe094e0 2026-03-23 Mon 05:43:57:108.843 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:57:108.851 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:57:109.107 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:109.122 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:109.186 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:109.204 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:109.214 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:109.217 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:109.249 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:43:57:109.276 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: SANDESH: Send FAILED: 1774244637109162 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:43:57:109.286 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:43:57:109.227 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:109.790 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:109.813 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:109.823 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:109.837 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:109.853 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:109.902 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:109.922 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:109.939 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:109.984 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:110.004 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:43:57:110.028 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: SANDESH: Send FAILED: 1774244637109888 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:43:57:110.037 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:43:57:110.422 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:110.451 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:110.461 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:110.474 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:110.483 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:110.495 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:110.505 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:110.522 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:110.531 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:110.545 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:110.554 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:110.567 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:110.576 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:110.588 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:110.597 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:110.609 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:110.623 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:110.642 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:110.657 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:110.675 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:57:110.689 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:57:110.721 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:111.225 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:111.241 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:43:57:111.268 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SANDESH: Send FAILED: 1774244637110753 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:43:57:111.276 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:43:57:111.292 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:111.312 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:112.162 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:112.185 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:112.210 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:112.762 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:57:112.773 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:43:57:112.794 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SANDESH: Send FAILED: 1774244637112231 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:43:57:112.801 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:43:57:112.818 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:112.834 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:112.848 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:112.867 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:57:113.429 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:43:57:113.660 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:43:57:113.677 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:43:57:113.705 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:43:57:113.832 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 4, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2026-03-23 Mon 05:43:57:113.857 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:43:57:113.881 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2026-03-23 Mon 05:43:57:113.892 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:43:57:113.899 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:43:57:113.907 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:43:57:113.933 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Established 2026-03-23 Mon 05:44:31:715.538 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-23 Mon 05:45:31:716.171 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-23 Mon 05:45:55:846.462 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe066a0 2026-03-23 Mon 05:45:55:846.515 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.587 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe05d60 2026-03-23 Mon 05:45:55:846.602 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.732 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe05420 2026-03-23 Mon 05:45:55:846.746 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.772 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe04ae0 2026-03-23 Mon 05:45:55:846.783 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.809 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:45:55:846.816 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe041a0 2026-03-23 Mon 05:45:55:846.825 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.996 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:45:55:847.027 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:45:55:847.042 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:45:55:847.055 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:45:55:847.066 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:45:55:886.088 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe03870 2026-03-23 Mon 05:45:55:886.139 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.276 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xdfc1d0 2026-03-23 Mon 05:45:55:886.292 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.315 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe03340 2026-03-23 Mon 05:45:55:886.324 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.343 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe02ca0 2026-03-23 Mon 05:45:55:886.361 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.471 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:45:55:886.480 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe025f0 2026-03-23 Mon 05:45:55:886.489 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.587 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:45:55:886.608 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:45:55:886.623 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:45:55:886.635 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:45:55:889.256 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:45:55:956.357 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe094e0 2026-03-23 Mon 05:45:55:956.416 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:956.524 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe08ba0 2026-03-23 Mon 05:45:55:956.539 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:956.561 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe08260 2026-03-23 Mon 05:45:55:956.571 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:956.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe07920 2026-03-23 Mon 05:45:55:956.599 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:956.620 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:45:55:956.628 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Disconnect status: -1 0xe06fe0 2026-03-23 Mon 05:45:55:956.636 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:956.837 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:45:55:956.866 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:45:55:956.880 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:45:55:956.892 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:45:55:956.905 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:45:56:026.745 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.50:38770::10.0.0.33:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2026-03-23 Mon 05:45:56:026.784 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.50:38770::10.0.0.33:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:45:56:026.805 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:38770::10.0.0.33:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-23 Mon 05:45:56:026.813 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:38770::10.0.0.33:8086(-1):EvTcpClose 2026-03-23 Mon 05:45:56:028.086 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvTcpClose in state Established 2026-03-23 Mon 05:45:56:028.099 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Established : EvTcpClose 2026-03-23 Mon 05:45:56:028.338 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:45:56:028.426 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:56:028.585 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244756028173 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = cn-jenkins-deploy-platform-ansible-os-5515-3. collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 137118 calls = 18 average_bytes = 7617 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-03-23 Mon 05:45:56:028.606 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:56:028.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:56:028.640 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244756028201 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:56:028.647 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:45:56:028.656 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2026-03-23 Mon 05:45:56:028.664 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:56:028.786 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244756028412 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 137118 calls = 18 average_bytes = 7617 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-03-23 Mon 05:45:56:028.801 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:56:031.738 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:52000::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:45:56:031.761 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:52000::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:45:56:031.769 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:52000::10.0.0.50:8086(-1):EvTcpConnected 2026-03-23 Mon 05:45:56:034.859 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:45:56:034.887 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:45:56:034.941 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: ClientInit 2026-03-23 Mon 05:45:56:034.969 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-2.:contrail-query-engine:0:Database count 2 2026-03-23 Mon 05:45:56:035.070 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:56:035.080 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:45:56:035.104 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: SANDESH: Send FAILED: 1774244756034925 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:56:035.112 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:45:56:035.121 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:56:035.128 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:45:56:035.146 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:56:035.152 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:45:56:035.286 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: SANDESH: Send FAILED: 1774244756035059 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 137118 calls = 18 average_bytes = 7617 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 15, _iter59->first = sent _iter59->second = 19, ] ] ] 2026-03-23 Mon 05:45:56:035.300 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:45:56:057.598 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:52000::10.0.0.50:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-23 Mon 05:45:56:057.626 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:52000::10.0.0.50:8086(-1):EvTcpClose 2026-03-23 Mon 05:45:56:057.900 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-23 Mon 05:45:56:057.918 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: ClientInit : EvTcpClose 2026-03-23 Mon 05:45:56:058.065 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Return to idle with 5000 2026-03-23 Mon 05:45:56:058.076 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Idle 2026-03-23 Mon 05:45:56:058.138 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:56:058.308 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: SANDESH: Send FAILED: 1774244756058012 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138691 calls = 19 average_bytes = 7299 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-03-23 Mon 05:45:56:058.326 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:56:058.337 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:56:058.358 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: SANDESH: Send FAILED: 1774244756058045 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:56:058.366 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:45:56:058.374 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:45:56:058.386 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:56:058.513 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: SANDESH: Send FAILED: 1774244756058127 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138691 calls = 19 average_bytes = 7299 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 17, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-03-23 Mon 05:45:56:058.525 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:00:847.082 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:00:847.262 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:00:847.320 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:00:847.347 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:00:847.358 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:00:847.387 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:00:847.413 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:00:847.464 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:00:847.480 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:00:847.493 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:00:886.651 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:00:886.789 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:00:886.850 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:00:886.885 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:00:887.109 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:00:887.208 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:00:887.228 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:00:887.241 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:00:889.304 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:00:889.588 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:00:956.906 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:00:957.041 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:00:957.091 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:00:957.123 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:00:957.163 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:00:957.246 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:00:957.279 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:00:957.292 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:00:957.306 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:00:957.320 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:46:01:058.484 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:01:058.693 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:46:01:058.794 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:01:058.828 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: SANDESH: Send FAILED: 1774244761058545 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:01:058.838 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:01:058.848 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:01:059.024 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: SANDESH: Send FAILED: 1774244761058780 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138691 calls = 19 average_bytes = 7299 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-03-23 Mon 05:46:01:059.042 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:01:061.017 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:53848::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:46:01:061.044 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:53848::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:46:01:061.052 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:53848::10.0.0.38:8086(-1):EvTcpConnected 2026-03-23 Mon 05:46:01:061.321 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:46:01:061.335 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:46:01:061.367 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: ClientInit 2026-03-23 Mon 05:46:01:061.379 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-2.:contrail-query-engine:0:Database count 3 2026-03-23 Mon 05:46:01:061.475 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:01:061.485 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:46:01:061.508 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: SANDESH: Send FAILED: 1774244761061360 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:01:061.516 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:46:01:061.524 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:01:061.531 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:46:01:061.542 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:01:061.548 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:46:01:061.727 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: SANDESH: Send FAILED: 1774244761061463 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 138691 calls = 19 average_bytes = 7299 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 22, _iter59->first = sent _iter59->second = 20, ] ] ] 2026-03-23 Mon 05:46:01:061.745 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659647903488, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:46:01:716.911 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659168110336, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:01:716.948 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659168110336, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:46:01:717.040 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659168110336, Pid 1]: SANDESH: Send FAILED: 1774244761716787 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 45 dequeues = 45 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 56 dequeues = 56 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1774244761061366 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1774244761061540 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:46:01:717.062 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659168110336, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:46:01:717.073 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659168110336, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:01:717.080 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659168110336, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:46:01:717.227 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659168110336, Pid 1]: SANDESH: Send FAILED: 1774244761716870 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 24, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:01:717.241 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659168110336, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:46:02:108.783 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:02:108.840 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: ClientInit : EvSandeshSend : QEDbStatsUve 2026-03-23 Mon 05:46:02:108.885 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: SANDESH: Send FAILED: 1774244762108727 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-23 Mon 05:46:02:108.906 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Received UVE message in wrong state : QEDbStatsUve 2026-03-23 Mon 05:46:03:061.525 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: 10.0.0.38:8086 EvConnectTimerExpired in state ClientInit 2026-03-23 Mon 05:46:03:061.593 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2026-03-23 Mon 05:46:03:061.604 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: ClientInit : EvConnectTimerExpired 2026-03-23 Mon 05:46:03:061.860 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Return to idle with 5000 2026-03-23 Mon 05:46:03:061.876 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Idle 2026-03-23 Mon 05:46:03:061.944 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:03:062.106 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: SANDESH: Send FAILED: 1774244763061730 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:03:062.132 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:03:062.146 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:03:062.168 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: SANDESH: Send FAILED: 1774244763061766 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:03:062.176 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:03:062.185 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:03:062.196 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:03:062.340 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: SANDESH: Send FAILED: 1774244763061931 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 27, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:03:062.360 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659660498688, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:05:847.424 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:05:847.640 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:05:847.724 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:05:847.764 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:05:847.780 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:05:847.825 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:05:847.867 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:05:847.922 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:05:847.945 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:05:847.975 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:05:887.215 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:05:887.353 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:05:887.388 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:05:887.418 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:05:887.626 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:05:887.687 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:05:887.703 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:05:887.716 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:05:889.645 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:05:889.943 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:05:957.339 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:05:957.507 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:05:957.557 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:05:957.604 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:05:957.644 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:05:957.715 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:05:957.752 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:05:957.773 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:05:957.792 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:05:957.809 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:46:08:062.497 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:08:062.715 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-23 Mon 05:46:08:062.836 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:08:062.921 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: SANDESH: Send FAILED: 1774244768062576 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:08:062.933 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:08:062.944 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:08:062.970 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:46:08:062.997 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:46:08:063.111 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: SANDESH: Send FAILED: 1774244768062819 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:08:063.135 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:08:063.146 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:46:08:063.154 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:46:08:063.251 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:08:063.280 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Idle 2026-03-23 Mon 05:46:08:063.340 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:08:063.484 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: SANDESH: Send FAILED: 1774244768063209 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:08:063.501 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:08:063.511 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:08:063.532 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: SANDESH: Send FAILED: 1774244768063230 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:08:063.539 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:08:063.548 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:08:063.567 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:08:063.750 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: SANDESH: Send FAILED: 1774244768063327 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:08:063.770 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:10:847.799 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:10:847.944 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:10:848.016 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:10:848.041 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:10:848.052 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:10:848.080 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:10:848.107 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:10:848.139 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:10:848.154 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:10:848.166 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:10:887.715 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:10:887.848 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:10:887.885 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:10:887.915 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:10:888.142 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:10:888.173 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:10:888.188 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:10:888.202 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:10:889.998 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:10:890.315 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:10:957.789 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:10:957.929 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:10:957.972 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:10:958.002 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:10:958.029 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:10:958.165 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:10:958.198 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:10:959.215 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:10:959.245 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:10:959.260 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:46:12:063.382 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:12:063.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:46:12:063.704 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:12:063.732 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: SANDESH: Send FAILED: 1774244772063450 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:12:063.741 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:12:063.751 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:12:063.844 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:46:12:063.875 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:46:12:063.896 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: SANDESH: Send FAILED: 1774244772063691 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:12:063.917 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:12:063.928 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:46:12:063.936 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:46:12:064.040 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:12:064.069 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Idle 2026-03-23 Mon 05:46:12:064.130 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:12:064.282 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: SANDESH: Send FAILED: 1774244772063998 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:12:064.299 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:12:064.309 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:12:064.330 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: SANDESH: Send FAILED: 1774244772064018 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:12:064.337 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:12:064.345 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:12:064.363 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:12:064.500 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: SANDESH: Send FAILED: 1774244772064117 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:12:064.515 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:15:848.084 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:15:848.245 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:15:848.296 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:15:848.322 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:15:848.333 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:15:848.363 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:15:848.390 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:15:848.422 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:15:848.437 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:15:848.449 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:15:888.207 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:15:888.344 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:15:888.392 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:15:888.421 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:15:888.695 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:15:888.750 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:15:888.765 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:15:888.778 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:15:890.352 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:15:890.629 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:15:958.262 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:15:958.560 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:15:959.192 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:15:959.257 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:15:959.292 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:15:959.321 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:15:959.358 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:15:959.381 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:15:959.395 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:15:959.409 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:46:20:064.619 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:20:064.871 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:46:20:065.030 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:20:065.071 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: SANDESH: Send FAILED: 1774244780064705 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:20:065.095 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:20:065.108 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:20:065.215 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:46:20:065.240 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:46:20:065.377 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: SANDESH: Send FAILED: 1774244780065008 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:20:065.402 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:20:065.417 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:46:20:065.427 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:46:20:065.565 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:20:065.612 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Idle 2026-03-23 Mon 05:46:20:065.704 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:20:065.929 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: SANDESH: Send FAILED: 1774244780065505 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 42, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:20:065.956 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:20:065.984 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:20:066.014 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: SANDESH: Send FAILED: 1774244780065534 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:20:066.030 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:20:066.043 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:20:066.072 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:20:066.318 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: SANDESH: Send FAILED: 1774244780065684 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 42, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:20:066.353 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:20:848.356 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:20:848.493 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:20:848.546 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:20:848.571 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:20:848.582 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:20:848.622 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:20:848.649 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:20:848.681 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:20:848.696 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:20:848.708 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:20:888.776 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:20:888.896 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:20:888.931 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:20:888.965 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:20:889.199 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:20:889.231 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:20:889.244 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:20:889.258 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:20:890.670 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:20:890.931 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:20:958.639 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:20:958.930 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:20:959.383 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:20:959.436 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:20:959.466 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:20:959.494 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:20:959.545 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:20:959.570 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:20:959.590 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:20:959.603 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:46:25:848.653 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:25:848.838 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:25:848.894 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:25:848.923 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:25:848.934 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:25:848.977 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:25:849.005 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:25:849.043 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:25:849.065 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:25:849.082 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:25:889.274 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:25:889.414 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:25:889.454 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:25:889.484 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:25:889.733 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:25:889.762 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:25:889.775 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:25:889.788 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:25:890.954 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:25:891.196 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:25:958.995 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:25:959.289 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:25:959.582 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:25:959.623 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:25:959.651 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:25:959.774 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:25:959.818 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:25:959.865 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:25:959.880 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:25:959.892 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:46:30:849.010 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:30:849.236 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:30:849.297 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:30:849.329 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:30:849.340 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:30:849.370 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:30:849.396 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:30:849.430 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:30:849.445 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:30:849.457 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:30:889.847 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:30:890.063 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:30:890.107 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:30:890.138 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:30:890.335 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:30:890.378 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:30:890.396 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:30:890.409 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:30:891.227 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:30:891.424 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:30:959.407 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:30:959.765 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:30:959.869 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:30:960.032 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:30:960.093 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:30:960.126 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:30:960.179 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:30:960.199 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:30:960.219 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:30:960.232 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:46:31:717.626 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-23 Mon 05:46:31:717.719 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:31:717.824 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: SANDESH: Send FAILED: 1774244791717478 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 66 dequeues = 66 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 88 dequeues = 88 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244780065610 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244780066069 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:46:31:717.844 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:31:717.862 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:31:718.026 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: SANDESH: Send FAILED: 1774244791717607 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:31:718.059 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659172308736, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:35:849.367 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:35:849.527 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:35:849.581 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:35:849.606 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:35:849.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:35:849.647 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:35:849.673 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:35:849.704 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:35:849.719 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:35:849.730 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:35:890.398 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:35:890.544 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:35:890.581 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:35:890.710 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:35:890.800 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:35:890.827 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:35:890.901 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:35:890.918 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:35:891.455 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:35:891.718 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:35:959.839 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:35:960.108 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:35:960.207 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:35:960.270 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:35:960.307 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:35:960.368 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:35:960.415 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:35:960.446 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:35:960.467 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:35:960.480 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:46:36:066.167 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:36:066.380 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-23 Mon 05:46:36:066.473 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:36:066.523 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244796066252 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:36:066.540 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:36:066.551 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:36:066.687 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244796066462 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 47, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:36:066.702 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:36:067.170 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:46:36:067.182 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:46:36:068.094 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:46:36:068.115 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:46:36:068.263 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:36:068.302 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Idle 2026-03-23 Mon 05:46:36:068.396 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:36:068.633 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244796068203 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 49, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:36:068.657 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:36:068.672 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:36:068.703 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244796068233 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:36:068.714 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:36:068.730 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:36:068.743 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:36:068.975 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244796068383 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 49, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:46:36:068.996 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:40:849.646 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:40:849.776 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:40:849.829 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:40:849.852 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:40:849.863 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:40:849.891 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:40:849.927 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:40:849.958 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:40:849.986 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:40:850.005 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:40:890.856 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:40:890.982 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:40:891.020 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:40:891.048 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:40:891.320 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:40:891.365 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:40:891.386 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:40:891.406 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:40:891.747 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:40:891.995 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:40:960.197 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:40:960.447 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:40:960.496 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:40:960.552 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:40:960.579 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:40:960.638 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:40:960.665 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:40:960.684 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:40:960.697 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:40:960.709 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:46:45:849.893 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:45:850.041 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:45:850.106 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:45:850.131 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:45:850.141 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:45:850.170 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:45:850.196 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:45:850.226 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:45:850.264 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:45:850.277 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:45:891.390 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:45:891.518 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:45:891.551 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:45:891.578 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:45:891.750 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:45:891.803 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:45:891.819 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:45:891.831 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:45:892.030 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:45:892.293 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:45:960.702 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:45:960.827 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:45:960.859 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:45:960.910 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:45:960.937 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:45:961.054 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:45:961.080 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:45:961.093 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:45:961.114 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:45:961.127 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:46:50:850.166 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:50:850.344 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:50:850.399 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:50:850.426 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:50:850.437 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:50:850.465 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:50:850.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:50:850.523 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:50:850.538 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:50:850.550 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:50:891.802 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:50:891.916 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:50:891.947 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:50:891.982 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:50:892.229 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:50:892.258 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:50:892.272 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:50:892.318 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:50:892.372 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:50:892.556 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:50:961.128 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:50:991.279 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:50:991.338 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:50:991.387 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:50:991.439 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:50:991.488 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:50:991.523 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:50:991.537 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:50:991.550 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:50:991.562 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:46:55:850.490 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:46:55:850.677 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:46:55:850.735 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:46:55:850.766 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:46:55:850.777 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:46:55:850.807 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:46:55:850.840 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:46:55:850.887 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:46:55:850.904 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:46:55:850.917 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:46:55:892.321 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:46:55:892.462 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:46:55:892.504 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:46:55:892.535 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:46:55:892.581 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:46:55:892.642 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:46:55:892.687 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:46:55:892.703 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:46:55:892.724 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:46:55:892.797 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:46:55:991.566 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:46:55:991.710 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:46:55:991.762 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:46:55:991.802 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:46:55:991.836 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:46:55:991.980 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:46:55:992.008 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:46:55:992.022 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:46:55:992.034 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:46:55:992.046 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:47:00:850.804 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:47:00:850.978 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:47:00:851.034 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:47:00:851.058 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:47:00:851.069 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:47:00:851.099 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:47:00:851.125 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:47:00:851.160 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:47:00:851.175 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:47:00:851.187 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:47:00:892.719 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:47:00:892.864 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:47:00:892.910 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:47:00:892.947 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:47:00:892.995 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:47:00:893.121 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:47:00:893.171 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:47:00:893.191 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:47:00:893.210 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:47:00:893.383 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:47:00:992.052 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:47:00:992.211 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:47:00:992.307 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:47:00:992.391 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:47:00:992.422 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:47:00:992.478 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:47:00:992.510 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:47:00:992.524 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:47:00:992.538 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:47:00:992.550 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:47:01:721.102 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:01:721.193 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SANDESH: Send FAILED: 1774244821720968 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 73 dequeues = 73 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 98 dequeues = 98 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244796068300 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244796068741 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:47:01:721.207 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:01:721.225 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:01:721.360 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SANDESH: Send FAILED: 1774244821721073 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:01:721.374 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:02:112.379 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:02:112.454 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: SANDESH: Send FAILED: 1774244822112349 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-23 Mon 05:47:02:112.465 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659664697088, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-03-23 Mon 05:47:05:851.129 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:47:05:851.349 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:47:05:851.420 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:47:05:851.452 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:47:05:851.464 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:47:05:851.495 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:47:05:851.522 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:47:05:851.559 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:47:05:851.574 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:47:05:851.586 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:47:05:893.238 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:47:05:893.463 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:47:05:893.531 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:47:05:893.582 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:47:05:893.613 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:47:05:893.776 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:47:05:893.874 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:47:05:893.899 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:47:05:893.923 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:47:05:893.944 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:47:05:992.558 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:47:05:992.703 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:47:05:992.747 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:47:05:992.795 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:47:05:992.832 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:47:05:992.893 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:47:05:992.943 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:47:05:992.958 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:47:05:992.984 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:47:05:992.998 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:47:06:069.730 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:47:06:070.141 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:47:06:070.363 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:06:070.416 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: SANDESH: Send FAILED: 1774244826069851 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:47:06:070.431 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:06:070.441 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:06:070.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: SANDESH: Send FAILED: 1774244826070335 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 55, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:06:070.616 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659681490688, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:06:071.343 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:47:06:071.354 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:47:06:071.846 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:47:06:071.857 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:47:06:072.087 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:47:06:072.128 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Idle 2026-03-23 Mon 05:47:06:072.190 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:06:072.370 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: SANDESH: Send FAILED: 1774244826072031 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:06:072.394 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:06:072.405 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:06:072.427 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: SANDESH: Send FAILED: 1774244826072060 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:47:06:072.435 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:06:072.443 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:47:06:072.453 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:06:072.575 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: SANDESH: Send FAILED: 1774244826072180 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:06:072.591 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:10:851.501 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:47:10:851.633 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:47:10:851.687 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe066a0 2026-03-23 Mon 05:47:10:851.709 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05d60 2026-03-23 Mon 05:47:10:851.720 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:47:10:851.749 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:47:10:851.775 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:47:10:851.807 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe05420 2026-03-23 Mon 05:47:10:851.821 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe04ae0 2026-03-23 Mon 05:47:10:851.833 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe041a0 2026-03-23 Mon 05:47:10:893.849 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:47:10:893.983 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:47:10:894.019 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:47:10:894.047 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:47:10:894.073 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:47:10:894.341 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03870 2026-03-23 Mon 05:47:10:894.385 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe02ca0 2026-03-23 Mon 05:47:10:894.401 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe03340 2026-03-23 Mon 05:47:10:894.414 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xdfc1d0 2026-03-23 Mon 05:47:10:894.428 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe025f0 2026-03-23 Mon 05:47:10:992.976 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:47:10:993.110 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:47:10:993.169 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:47:10:993.198 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:47:10:993.226 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:47:10:993.310 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe094e0 2026-03-23 Mon 05:47:10:993.339 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08ba0 2026-03-23 Mon 05:47:10:993.353 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe08260 2026-03-23 Mon 05:47:10:993.366 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe07920 2026-03-23 Mon 05:47:10:993.379 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: -1 0xe06fe0 2026-03-23 Mon 05:47:15:851.756 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe066a0 2026-03-23 Mon 05:47:15:851.945 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05d60 2026-03-23 Mon 05:47:15:852.012 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe066a0 2026-03-23 Mon 05:47:15:852.020 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:852.044 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:852.055 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe05d60 2026-03-23 Mon 05:47:15:852.061 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:852.069 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:852.076 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe05420 2026-03-23 Mon 05:47:15:852.120 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe04ae0 2026-03-23 Mon 05:47:15:852.159 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe041a0 2026-03-23 Mon 05:47:15:852.239 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe05420 2026-03-23 Mon 05:47:15:852.247 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:852.257 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:852.275 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe04ae0 2026-03-23 Mon 05:47:15:852.281 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:852.289 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:852.297 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe041a0 2026-03-23 Mon 05:47:15:852.302 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:852.310 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:854.393 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:854.416 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:854.430 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:854.439 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:854.452 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:854.461 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:854.473 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:854.482 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:854.494 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:854.503 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:854.530 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:855.039 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:855.063 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:855.085 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:15:855.096 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:855.254 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:855.263 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: SANDESH: Send FAILED: 1774244835854560 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:15:855.284 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:15:894.435 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03870 2026-03-23 Mon 05:47:15:894.593 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe02ca0 2026-03-23 Mon 05:47:15:894.641 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe03340 2026-03-23 Mon 05:47:15:894.679 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xdfc1d0 2026-03-23 Mon 05:47:15:894.704 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe025f0 2026-03-23 Mon 05:47:15:895.088 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe02ca0 2026-03-23 Mon 05:47:15:895.116 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:895.138 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:895.151 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe03870 2026-03-23 Mon 05:47:15:895.157 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:895.167 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:895.176 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xdfc1d0 2026-03-23 Mon 05:47:15:895.181 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:895.189 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:895.209 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe03340 2026-03-23 Mon 05:47:15:895.214 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:895.222 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:895.230 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe025f0 2026-03-23 Mon 05:47:15:895.235 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:895.243 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:895.819 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:895.839 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:895.853 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:895.863 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:895.917 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:895.929 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:895.961 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:896.051 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:15:896.081 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: SANDESH: Send FAILED: 1774244835895998 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:15:896.090 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659668895488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:15:896.590 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:896.620 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:896.638 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:896.648 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:896.666 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:896.680 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:896.697 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:896.711 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:993.382 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe094e0 2026-03-23 Mon 05:47:15:993.490 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08ba0 2026-03-23 Mon 05:47:15:993.528 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe08260 2026-03-23 Mon 05:47:15:993.556 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe07920 2026-03-23 Mon 05:47:15:993.583 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0xe06fe0 2026-03-23 Mon 05:47:15:993.696 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe094e0 2026-03-23 Mon 05:47:15:993.706 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:993.743 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:993.756 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe08ba0 2026-03-23 Mon 05:47:15:993.762 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:993.771 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:993.779 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe08260 2026-03-23 Mon 05:47:15:993.785 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:993.802 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:993.810 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe07920 2026-03-23 Mon 05:47:15:993.815 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:993.823 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:993.849 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: RAC_Connect status: 0 0xe06fe0 2026-03-23 Mon 05:47:15:993.855 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:993.863 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:994.084 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:994.117 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:994.133 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:994.143 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:994.166 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:994.485 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659685689088, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:15:994.522 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659685689088, Pid 1]: SANDESH: Send FAILED: 1774244835994207 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:15:994.532 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659685689088, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:15:994.552 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:994.569 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:994.584 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:994.599 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:994.608 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:994.621 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:994.630 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:994.647 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:994.661 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:994.675 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:31:722.958 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-23 Mon 05:47:31:723.049 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:31:723.137 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244851722816 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 112 dequeues = 112 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244826072127 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244835994478 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:47:31:723.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:31:723.178 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:31:723.323 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244851722937 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 150, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:31:723.338 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:36:074.736 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:47:36:074.934 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:47:36:075.031 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:36:075.059 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: SANDESH: Send FAILED: 1774244856074814 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:36:075.077 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:36:075.087 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:36:075.269 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: SANDESH: Send FAILED: 1774244856075019 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 150, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 65, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:36:075.287 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659702482688, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:36:080.432 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:54936::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:47:36:080.461 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:54936::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:47:36:080.469 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659846154368, Pid 1]: 10.0.0.50:54936::10.0.0.38:8086(-1):EvTcpConnected 2026-03-23 Mon 05:47:36:083.095 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:47:36:083.108 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:47:36:083.149 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: ClientInit 2026-03-23 Mon 05:47:36:083.161 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-2.:contrail-query-engine:0:Database count 4 2026-03-23 Mon 05:47:36:083.246 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:47:36:083.255 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:47:36:083.278 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244856083142 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:36:083.286 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:47:36:083.294 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:47:36:083.301 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:47:36:083.313 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:47:36:083.319 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:47:36:083.450 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: SANDESH: Send FAILED: 1774244856083236 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 67, _iter59->first = sent _iter59->second = 21, ] ] ] 2026-03-23 Mon 05:47:36:083.469 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659698284288, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:47:36:183.055 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:47:36:183.128 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:47:36:183.140 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:47:36:183.166 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:47:36:183.390 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 1016 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 140264 calls = 20 average_bytes = 7013 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 112 dequeues = 112 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244826072127 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244835994478 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 67, _iter59->first = sent _iter59->second = 21, ] ] max_sm_queue_count = 5 proxy seq 48 2026-03-23 Mon 05:47:36:183.423 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:47:36:183.447 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 34 2026-03-23 Mon 05:47:36:183.459 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:47:36:183.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.143306 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 4 2026-03-23 Mon 05:47:36:183.501 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:47:36:183.514 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:47:36:183.545 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659656300288, Pid 1]: Established 2026-03-23 Mon 05:47:41:827.203 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: OnMessage control in state: Established 2026-03-23 Mon 05:47:41:827.260 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-03-23 Mon 05:47:41:827.270 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Established : EvSandeshMessageRecv 2026-03-23 Mon 05:47:41:827.311 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: Received Ctrl Message with size 3 2026-03-23 Mon 05:47:41:827.325 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 49, total = 0 2026-03-23 Mon 05:47:41:827.337 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 36, total = 0 2026-03-23 Mon 05:47:41:827.346 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 4, total = 0 2026-03-23 Mon 05:47:41:827.353 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659689887488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:48:31:723.559 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659694085888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-23 Mon 05:49:31:725.272 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659673093888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-23 Mon 05:50:31:726.287 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659673093888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-23 Mon 05:51:31:727.458 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659677292288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-23 Mon 05:51:45:238.004 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659677292288, Pid 1]: Processing scm::EvStop in state Established 2026-03-23 Mon 05:51:45:238.083 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659677292288, Pid 1]: Idle 2026-03-23 Mon 05:51:45:238.182 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659677292288, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:51:45:238.196 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659677292288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:51:45:238.392 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659677292288, Pid 1]: SANDESH: Send FAILED: 1774245105238167 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244611714921 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 3463 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 388664 calls = 44 average_bytes = 8833 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 28 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 69, _iter59->first = sent _iter59->second = 48, ] ] ] 2026-03-23 Mon 05:51:45:238.418 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140659677292288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:52:09:983.923 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: http-server-port 8091 2026-03-23 Mon 05:52:09:983.987 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Max-tasks 16 2026-03-23 Mon 05:52:09:984.004 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Max-slice 100 2026-03-23 Mon 05:52:09:984.010 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Collectors 10.0.0.38:8086 2026-03-23 Mon 05:52:09:984.015 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Collectors 10.0.0.50:8086 2026-03-23 Mon 05:52:09:984.020 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Collectors 10.0.0.33:8086 2026-03-23 Mon 05:52:09:984.040 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: ROLE : Generator 2026-03-23 Mon 05:52:09:984.046 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-23 Mon 05:52:09:984.050 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5515-2. 2026-03-23 Mon 05:52:09:984.055 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-23 Mon 05:52:09:984.060 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-23 Mon 05:52:09:984.064 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-23 Mon 05:52:09:984.070 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-23 Mon 05:52:09:996.305 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: HTTP Introspect Init 2026-03-23 Mon 05:52:09:996.427 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 138 2026-03-23 Mon 05:52:09:996.524 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Sandesh Http Server Port: 8091 2026-03-23 Mon 05:52:09:996.582 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-23 Mon 05:52:09:996.590 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-23 Mon 05:52:09:996.921 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Idle 2026-03-23 Mon 05:52:09:997.066 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: No Client: 1774245129997012 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245129996932 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:52:09:997.204 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-23 Mon 05:52:09:997.220 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-23 Mon 05:52:09:997.231 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-03-23 Mon 05:52:09:997.349 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Processing scm::EvStart in state Idle 2026-03-23 Mon 05:52:09:997.428 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Disconnect 2026-03-23 Mon 05:52:09:997.488 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-23 Mon 05:52:09:997.676 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:52:09:997.730 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:52:09:997.753 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: SANDESH: Send FAILED: 1774245129997421 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:52:09:997.761 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:52:09:997.770 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:52:09:997.830 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: SANDESH: Send FAILED: 1774245129997477 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245129996932 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:52:09:997.840 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:52:09:997.851 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:52:09:997.896 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: SANDESH: Send FAILED: 1774245129997517 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245129996932 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:52:09:997.905 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:52:09:997.913 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:52:09:997.924 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: SANDESH: Send FAILED: 1774245129997533 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:52:09:997.933 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:52:09:997.940 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:52:09:997.978 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: SANDESH: Send FAILED: 1774245129997720 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245129996932 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:52:09:997.987 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041380963904, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:52:10:063.178 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-23 Mon 05:52:10:068.380 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:49532::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2026-03-23 Mon 05:52:10:068.415 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: 10.0.0.50:49532::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:52:10:068.423 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: 10.0.0.50:49532::10.0.0.38:8086(-1):EvTcpConnected 2026-03-23 Mon 05:52:10:068.447 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc3430 2026-03-23 Mon 05:52:10:068.452 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.462 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:068.474 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc3ff0 2026-03-23 Mon 05:52:10:068.479 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.486 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:068.496 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc4c30 2026-03-23 Mon 05:52:10:068.500 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.507 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:068.516 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc5870 2026-03-23 Mon 05:52:10:068.521 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.528 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:068.536 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc64b0 2026-03-23 Mon 05:52:10:068.541 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.548 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:068.556 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcbb540 2026-03-23 Mon 05:52:10:068.561 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.568 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:068.578 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc0540 2026-03-23 Mon 05:52:10:068.582 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:068.598 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc0df0 2026-03-23 Mon 05:52:10:068.605 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.612 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:068.621 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc2770 2026-03-23 Mon 05:52:10:068.626 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.632 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:068.641 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc1ab0 2026-03-23 Mon 05:52:10:068.646 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.653 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:068.661 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc70f0 2026-03-23 Mon 05:52:10:068.666 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.672 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:068.681 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc7d30 2026-03-23 Mon 05:52:10:068.976 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:068.990 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:069.010 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc8970 2026-03-23 Mon 05:52:10:069.016 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:069.022 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:069.032 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcc95b0 2026-03-23 Mon 05:52:10:069.037 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:069.043 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:069.053 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: RAC_Connect status: 0 0xcca1f0 2026-03-23 Mon 05:52:10:069.057 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:52:10:069.064 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:52:10:069.260 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:069.270 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:069.282 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:069.290 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:069.328 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:069.354 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:069.366 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:069.374 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:069.385 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:069.392 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:069.402 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:069.410 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:069.423 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:069.435 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:069.446 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:069.477 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:069.486 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:069.501 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:069.508 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:069.522 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:069.543 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:069.554 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:069.562 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:069.572 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:069.579 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:069.590 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:069.597 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:069.609 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:069.620 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:069.631 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:068.965 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:52:10:069.682 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: SANDESH: Send FAILED: 1774245130066872 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:52:10:069.690 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:52:10:069.702 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:52:10:069.709 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:52:10:069.728 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: ClientInit 2026-03-23 Mon 05:52:10:069.745 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-2.:contrail-query-engine:0:Database count 1 2026-03-23 Mon 05:52:10:069.961 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:52:10:069.975 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:52:10:069.993 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: SANDESH: Send FAILED: 1774245130069320 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:52:10:070.009 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:52:10:070.017 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:52:10:070.023 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:52:10:070.035 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: SANDESH: Send FAILED: 1774245130069349 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:52:10:070.043 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:52:10:070.050 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:52:10:070.056 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:52:10:070.067 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: SANDESH: Send FAILED: 1774245130069538 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:52:10:070.077 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:52:10:070.084 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:52:10:070.090 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:52:10:070.100 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: SANDESH: Send FAILED: 1774245130069724 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:52:10:070.106 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:52:10:070.112 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:52:10:070.118 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:52:10:070.136 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:52:10:070.142 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:52:10:070.223 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: SANDESH: Send FAILED: 1774245130069839 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774245129996932 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 46 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 48, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:52:10:070.256 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:52:10:072.724 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:072.780 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:072.796 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:072.805 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:072.823 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:076.296 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:076.328 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:076.336 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:076.349 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:076.356 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:076.368 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:52:10:076.375 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:52:10:076.382 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:52:10:076.412 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:52:10:076.431 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: SANDESH: Send FAILED: 1774245130072842 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:52:10:076.443 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:52:10:076.390 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:076.484 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:076.529 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:52:10:078.917 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:52:10:078.953 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:52:10:078.961 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:52:10:078.977 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:52:10:079.072 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774245129996932 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.50:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 46 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 48, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2026-03-23 Mon 05:52:10:079.098 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:52:10:079.118 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2026-03-23 Mon 05:52:10:079.129 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:52:10:079.137 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:52:10:079.143 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:52:10:079.170 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: Established 2026-03-23 Mon 05:53:09:997.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040558601792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-23 Mon 05:54:09:998.876 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-23 Mon 05:55:09:999.336 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-23 Mon 05:56:09:999.855 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041256252992, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-23 Mon 05:57:10:000.609 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040558601792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-23 Mon 05:58:10:001.131 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041252054592, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-23 Mon 05:59:10:001.684 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041256252992, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-23 Mon 06:00:10:002.137 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-23 Mon 06:01:10:002.594 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-03-23 Mon 06:02:10:003.063 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041256252992, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2026-03-23 Mon 06:03:10:003.889 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2026-03-23 Mon 06:04:10:004.408 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041376765504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2026-03-23 Mon 06:05:10:004.965 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2026-03-23 Mon 06:06:10:005.588 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041243657792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2026-03-23 Mon 06:07:10:006.156 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041247856192, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2026-03-23 Mon 06:07:52:119.889 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:55806 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:07:57:140.961 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:55812 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:08:10:006.724 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040558601792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2026-03-23 Mon 06:09:10:007.916 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041372567104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2026-03-23 Mon 06:09:28:831.321 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.38:35326 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:09:45:752.542 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.38 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-03-23 Mon 06:09:45:753.634 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID e3f64800-267e-11f1-8c20-00000a000026 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-03-23 Mon 06:09:45:753.696 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QEQueryLog: e3f64800-267e-11f1-8c20-00000a000026 :0:json query is: { end_time: 1774246185690472 , enqueue_time: 1774246185751975 , query_metadata: {"enqueue_time": 1774246185751853} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "a63dda54-26d6-4035-ab8b-b822522507fe", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:45:753.900 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: Chunks: 2 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-03-23 Mon 06:09:45:753.917 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-03-23 Mon 06:09:45:753.962 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for e3f64800-267e-11f1-8c20-00000a000026 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-03-23 Mon 06:09:45:754.058 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QEQueryLog: e3f64800-267e-11f1-8c20-00000a000026 :0:json query is: { end_time: 1774246185690472 , enqueue_time: 1774246185751975 , query_metadata: {"enqueue_time": 1774246185751853} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "a63dda54-26d6-4035-ab8b-b822522507fe", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:45:792.053 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:45:792.089 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:45:792.107 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:45:792.114 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:45:792.121 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 UNION between tables of sizes 0 and 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:45:792.145 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:45:792.153 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:45:792.162 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:45:792.169 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:45:792.177 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:45:792.322 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QEQueryLog: e3f64800-267e-11f1-8c20-00000a000026 :0:json query is: { end_time: 1774246185690472 , enqueue_time: 1774246185751975 , query_metadata: {"enqueue_time": 1774246185751853} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "a63dda54-26d6-4035-ab8b-b822522507fe", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:45:792.510 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QEQueryLog: e3f64800-267e-11f1-8c20-00000a000026 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id a63dda54-26d6-4035-ab8b-b822522507fe src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:45:792.616 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QEQueryLog: e3f64800-267e-11f1-8c20-00000a000026 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id a63dda54-26d6-4035-ab8b-b822522507fe src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:45:792.713 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QEQueryLog: e3f64800-267e-11f1-8c20-00000a000026 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id a63dda54-26d6-4035-ab8b-b822522507fe src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:45:792.746 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 UNION between tables of sizes 0 and 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:45:792.778 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:45:792.787 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:45:792.795 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:45:792.803 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:45:792.811 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:45:792.818 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:45:792.825 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:45:792.832 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:45:792.840 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: e3f64800-267e-11f1-8c20-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:45:793.038 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QEQueryLog: e3f64800-267e-11f1-8c20-00000a000026 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id a63dda54-26d6-4035-ab8b-b822522507fe src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:45:793.137 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QEQueryLog: e3f64800-267e-11f1-8c20-00000a000026 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id a63dda54-26d6-4035-ab8b-b822522507fe src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:45:793.274 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-03-23 Mon 06:09:45:793.334 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-03-23 Mon 06:09:45:793.378 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-03-23 Mon 06:09:45:793.708 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: Finished: QID e3f64800-267e-11f1-8c20-00000a000026 Table ObjectVMTable Time(ms) 40 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-03-23 Mon 06:09:45:793.752 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-03-23 Mon 06:09:45:793.946 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5515-2. table = ObjectVMTable query_stats= [ qid = e3f64800-267e-11f1-8c20-00000a000026 where = [[{"name": "ObjectId", "value": "a63dda54-26d6-4035-ab8b-b822522507fe", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 2 chunk_where_time = 37,0, 38,0, chunk_select_time = 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 40 rows = 5 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-03-23 Mon 06:09:48:810.816 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.38 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-03-23 Mon 06:09:48:811.438 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID e5c8fdac-267e-11f1-ae76-00000a000026 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-03-23 Mon 06:09:48:811.473 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QEQueryLog: e5c8fdac-267e-11f1-ae76-00000a000026 :0:json query is: { end_time: 1774246188799790 , enqueue_time: 1774246188810318 , query_metadata: {"enqueue_time": 1774246188810228} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333:ctest-vn22-62662333", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:48:811.603 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: Chunks: 2 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-03-23 Mon 06:09:48:811.618 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-03-23 Mon 06:09:48:811.656 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for e5c8fdac-267e-11f1-ae76-00000a000026 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-03-23 Mon 06:09:48:811.841 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041247856192, Pid 1]: [SYS_INFO]: QEQueryLog: e5c8fdac-267e-11f1-ae76-00000a000026 :0:json query is: { end_time: 1774246188799790 , enqueue_time: 1774246188810318 , query_metadata: {"enqueue_time": 1774246188810228} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333:ctest-vn22-62662333", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:48:847.885 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:48:847.923 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:48:847.932 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:48:847.942 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:48:847.952 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:48:847.961 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:48:847.971 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 UNION between tables of sizes 0 and 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:48:847.984 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:48:847.995 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:48:848.006 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:48:848.349 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: [SYS_INFO]: QEQueryLog: e5c8fdac-267e-11f1-ae76-00000a000026 ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:48:863.494 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:48:863.547 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:48:863.555 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:48:863.562 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:48:863.569 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:48:863.576 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:48:863.583 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:48:863.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:48:863.596 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:48:863.602 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: [SYS_INFO]: QELog: e5c8fdac-267e-11f1-ae76-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:48:863.778 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041247856192, Pid 1]: [SYS_INFO]: QEQueryLog: e5c8fdac-267e-11f1-ae76-00000a000026 :0:json query is: { end_time: 1774246188799790 , enqueue_time: 1774246188810318 , query_metadata: {"enqueue_time": 1774246188810228} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333:ctest-vn22-62662333", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:48:864.048 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-03-23 Mon 06:09:48:864.148 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-03-23 Mon 06:09:48:864.271 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-03-23 Mon 06:09:48:865.399 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041247856192, Pid 1]: [SYS_INFO]: QELog: Finished: QID e5c8fdac-267e-11f1-ae76-00000a000026 Table ObjectRoutingInstance Time(ms) 53 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-03-23 Mon 06:09:48:865.422 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040562800192, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5515-2. table = ObjectRoutingInstance query_stats= [ qid = e5c8fdac-267e-11f1-ae76-00000a000026 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333:ctest-vn22-62662333", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 2 chunk_where_time = 36,0, 49,0, chunk_select_time = 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 53 rows = 1 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-03-23 Mon 06:09:48:865.446 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041247856192, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-03-23 Mon 06:10:10:008.572 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040558601792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2026-03-23 Mon 06:11:10:009.113 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041256252992, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2026-03-23 Mon 06:12:10:009.889 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041376765504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2026-03-23 Mon 06:13:10:010.499 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041292600896, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2026-03-23 Mon 06:14:10:011.255 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040550204992, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2026-03-23 Mon 06:15:10:011.749 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041239459392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2026-03-23 Mon 06:16:10:012.795 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041300997696, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2026-03-23 Mon 06:17:10:013.286 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140040554403392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2026-03-23 Mon 06:18:10:013.819 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041247856192, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2026-03-23 Mon 06:19:10:014.360 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041376765504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2026-03-23 Mon 06:20:10:014.860 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041376765504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2026-03-23 Mon 06:21:07:981.468 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.38:36120 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:21:10:015.381 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041243657792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2026-03-23 Mon 06:22:10:015.942 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041260451392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2026-03-23 Mon 06:23:10:016.516 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041376765504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2026-03-23 Mon 06:23:41:953.606 UTC cn-jenkins-deploy-platform-ansible-os-5515-2 [Thread 140041388103808, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.50:8091::10.0.0.50:49544 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256