2026-03-30 Mon 03:36:38:036.609 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: http-server-port 8091 2026-03-30 Mon 03:36:38:036.749 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Max-tasks 16 2026-03-30 Mon 03:36:38:036.766 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Max-slice 100 2026-03-30 Mon 03:36:38:036.775 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Collectors 10.0.0.249:8086 2026-03-30 Mon 03:36:38:036.782 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Collectors 10.0.0.241:8086 2026-03-30 Mon 03:36:38:036.789 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Collectors 10.0.0.141:8086 2026-03-30 Mon 03:36:38:036.817 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: ROLE : Generator 2026-03-30 Mon 03:36:38:036.828 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-30 Mon 03:36:38:036.835 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5563-1. 2026-03-30 Mon 03:36:38:036.841 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-30 Mon 03:36:38:036.848 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-30 Mon 03:36:38:036.855 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-30 Mon 03:36:38:036.862 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-30 Mon 03:36:38:037.662 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: HTTP Introspect Init 2026-03-30 Mon 03:36:38:037.802 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-03-30 Mon 03:36:38:037.879 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Sandesh Http Server Port: 8091 2026-03-30 Mon 03:36:38:037.950 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-30 Mon 03:36:38:037.961 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-30 Mon 03:36:38:038.062 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Idle 2026-03-30 Mon 03:36:38:038.274 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: No Client: 1774841798038174 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:36:38:038.449 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-30 Mon 03:36:38:038.468 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-30 Mon 03:36:38:038.483 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Cassandra Servers: 10.0.0.249:9042 10.0.0.241:9042 10.0.0.141:9042 2026-03-30 Mon 03:36:38:045.209 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Processing scm::EvStart in state Idle 2026-03-30 Mon 03:36:38:045.293 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Disconnect 2026-03-30 Mon 03:36:38:045.398 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-30 Mon 03:36:38:045.677 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Connect : Start Connect timer 10.0.0.241:8086 2026-03-30 Mon 03:36:38:045.781 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:38:046.074 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: SANDESH: Send FAILED: 1774841798045282 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:36:38:046.093 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:36:38:046.108 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:38:046.243 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: SANDESH: Send FAILED: 1774841798045383 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:36:38:047.631 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:36:38:047.658 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:38:047.809 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: SANDESH: Send FAILED: 1774841798045470 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:36:38:047.833 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:36:38:047.847 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:38:047.871 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: SANDESH: Send FAILED: 1774841798045500 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:36:38:047.881 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:36:38:047.906 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:38:048.018 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: SANDESH: Send FAILED: 1774841798045764 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:36:38:048.035 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:36:38:050.797 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-30 Mon 03:36:38:050.825 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:36:38:050.917 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:36:38:050.992 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:36:38:051.034 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SANDESH: Send FAILED: 1774841798050867 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:36:38:051.045 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:36:43:056.492 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-30 Mon 03:36:43:056.536 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:36:43:056.581 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:36:48:068.482 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-30 Mon 03:36:48:068.539 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:36:48:068.568 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:36:53:071.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-30 Mon 03:36:53:071.838 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:36:53:071.870 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:36:58:080.421 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-30 Mon 03:36:58:080.484 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:36:58:080.539 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:37:03:144.598 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-30 Mon 03:37:03:144.678 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:37:03:144.729 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:37:08:176.035 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-30 Mon 03:37:08:176.087 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:37:08:176.115 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:37:13:286.041 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-30 Mon 03:37:13:286.085 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-30 Mon 03:37:13:286.107 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:37:18:321.104 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-30 Mon 03:37:18:321.171 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-30 Mon 03:37:23:379.616 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-30 Mon 03:37:28:395.536 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:37:28:395.619 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: SANDESH: Send FAILED: 1774841848395439 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:37:28:395.634 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:28:395.553 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c24830 2026-03-30 Mon 03:37:28:395.822 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:395.837 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:395.850 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c20da0 2026-03-30 Mon 03:37:28:395.855 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:395.864 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:395.872 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c25400 2026-03-30 Mon 03:37:28:395.877 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:395.884 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:395.904 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c25b50 2026-03-30 Mon 03:37:28:395.916 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:395.924 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:395.931 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c26470 2026-03-30 Mon 03:37:28:395.936 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:395.943 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:395.950 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c27ff0 2026-03-30 Mon 03:37:28:395.955 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:395.963 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:395.970 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c276b0 2026-03-30 Mon 03:37:28:395.975 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:395.982 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:395.989 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c26d70 2026-03-30 Mon 03:37:28:395.994 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:396.001 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:396.010 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c29270 2026-03-30 Mon 03:37:28:396.015 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:396.022 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:396.029 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c28930 2026-03-30 Mon 03:37:28:396.034 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:396.041 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:396.049 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c29bb0 2026-03-30 Mon 03:37:28:396.054 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:396.061 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:396.074 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2a4f0 2026-03-30 Mon 03:37:28:396.079 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:396.086 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:396.094 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2b770 2026-03-30 Mon 03:37:28:396.099 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:396.106 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:396.113 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2c0b0 2026-03-30 Mon 03:37:28:396.118 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:396.128 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:396.135 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2ae30 2026-03-30 Mon 03:37:28:396.140 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:37:28:396.147 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:37:28:396.172 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:37:28:396.179 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:37:28:396.227 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:37:28:396.262 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:37:28:396.412 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:37:28:396.452 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Idle 2026-03-30 Mon 03:37:28:396.515 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:28:397.434 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.453 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.466 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.475 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.491 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.500 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.511 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.520 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.532 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.540 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.551 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.560 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.571 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.580 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.590 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.599 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.610 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.618 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.628 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.637 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.650 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.658 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.669 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.677 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.688 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.696 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.706 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.715 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.725 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:37:28:397.733 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:37:28:397.773 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:397.869 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:397.898 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:397.914 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:397.929 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:397.952 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:397.976 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:397.991 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:398.008 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:398.021 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:398.039 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:398.060 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:398.073 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:398.085 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:398.097 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:37:28:398.226 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774841848396341 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 42 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:37:28:398.244 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:37:28:398.256 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:28:398.277 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774841848396369 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:37:28:398.284 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:28:398.292 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:37:28:398.330 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:28:398.436 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774841848396505 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 42 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 44, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:37:28:398.448 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:37:28:398.459 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:28:398.478 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774841848397764 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:37:28:398.484 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:28:398.492 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:28:398.508 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774841848397862 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.241:6379::Down.10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:37:28:398.514 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:28:398.521 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:28:398.537 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774841848397970 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:37:28:398.542 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:28:398.549 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:28:398.567 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774841848398054 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:37:28:398.573 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:28:398.661 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:28:398.704 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774841848398593 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774841848396451 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774841848398548 ] max_sm_queue_count = 7 ] 2026-03-30 Mon 03:37:28:398.710 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:37:28:398.718 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:37:28:398.812 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774841848398648 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 14, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:37:28:398.824 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:37:29:396.595 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:37:29:397.702 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Connect : Start Connect timer 10.0.0.141:8086 2026-03-30 Mon 03:37:29:397.864 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:37:29:397.948 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774841849396693 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:37:29:397.962 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:37:29:397.976 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:37:29:400.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:54246::10.0.0.141:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:37:29:400.807 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:54246::10.0.0.141:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:37:29:400.815 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:54246::10.0.0.141:8086(-1):EvTcpConnected 2026-03-30 Mon 03:37:29:400.929 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774841849397847 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:37:29:400.950 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:37:29:400.962 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:37:29:400.969 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:37:29:401.000 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: ClientInit 2026-03-30 Mon 03:37:29:401.032 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-1.:contrail-query-engine:0:Database count 1 2026-03-30 Mon 03:37:29:401.110 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:37:29:401.118 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:37:29:401.140 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774841849400994 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:37:29:401.150 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:37:29:401.159 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:37:29:401.165 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:37:29:401.176 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:37:29:401.182 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:37:29:401.287 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774841849401100 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:37:29:401.299 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:37:29:404.070 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: OnMessage control in state: ClientInit 2026-03-30 Mon 03:37:29:408.828 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-30 Mon 03:37:29:408.859 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-30 Mon 03:37:29:408.901 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Received Ctrl Message with size 0 2026-03-30 Mon 03:37:29:409.057 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774841848396451 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774841848398548 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 7 proxy seq 10 2026-03-30 Mon 03:37:29:409.087 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:37:29:409.112 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2026-03-30 Mon 03:37:29:409.123 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:37:29:409.130 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:37:29:409.137 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:37:29:409.168 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Established 2026-03-30 Mon 03:37:35:582.953 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706289350400, Pid 1]: OnMessage control in state: Established 2026-03-30 Mon 03:37:35:584.842 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-03-30 Mon 03:37:35:584.868 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: Established : EvSandeshMessageRecv 2026-03-30 Mon 03:37:35:584.932 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: Received Ctrl Message with size 3 2026-03-30 Mon 03:37:35:584.960 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2026-03-30 Mon 03:37:35:584.971 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2026-03-30 Mon 03:37:35:584.983 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2026-03-30 Mon 03:37:35:584.991 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:37:58:398.923 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-30 Mon 03:38:58:399.532 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706289350400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-30 Mon 03:39:58:400.355 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-30 Mon 03:40:36:876.962 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:8091::10.0.0.249:43834 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-30 Mon 03:40:56:853.534 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:8091::10.0.0.249:53888 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-30 Mon 03:40:58:401.016 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-30 Mon 03:41:58:402.017 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-30 Mon 03:42:58:403.701 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-30 Mon 03:43:19:097.323 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-30 Mon 03:43:19:097.369 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c26d70 2026-03-30 Mon 03:43:19:097.382 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:097.422 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c28930 2026-03-30 Mon 03:43:19:097.432 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:097.562 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c29270 2026-03-30 Mon 03:43:19:097.575 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:097.597 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c27ff0 2026-03-30 Mon 03:43:19:097.606 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:097.624 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c276b0 2026-03-30 Mon 03:43:19:097.632 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:097.790 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:43:19:097.821 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:43:19:098.643 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:43:19:098.679 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:43:19:098.699 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:43:19:434.749 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c2ae30 2026-03-30 Mon 03:43:19:434.816 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:434.981 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c2b770 2026-03-30 Mon 03:43:19:435.000 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:435.045 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c2c0b0 2026-03-30 Mon 03:43:19:435.060 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:435.091 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-30 Mon 03:43:19:435.101 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c29bb0 2026-03-30 Mon 03:43:19:435.113 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:435.138 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c2a4f0 2026-03-30 Mon 03:43:19:435.151 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:43:19:435.403 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:43:19:435.431 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:43:19:435.451 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:43:19:435.821 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:43:19:435.859 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:43:24:097.890 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:43:24:098.053 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:43:24:098.394 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:43:24:098.439 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:43:24:098.703 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:43:24:098.766 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:43:24:098.814 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:43:24:099.019 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:43:24:099.115 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:43:24:099.134 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:43:24:435.552 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:43:24:435.902 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:43:24:436.022 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:43:24:436.122 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:43:24:436.205 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:43:24:436.352 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:43:24:436.468 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:43:24:436.514 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:43:24:436.556 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:43:24:436.597 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:43:29:098.475 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:43:29:098.591 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:43:29:099.027 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:43:29:099.149 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:43:29:099.180 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:43:29:099.291 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:43:29:099.351 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:43:29:099.693 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:43:29:099.743 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:43:29:099.832 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:43:29:436.509 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:43:29:436.634 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:43:29:436.689 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:43:29:436.719 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:43:29:436.748 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:43:29:437.708 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:43:29:437.752 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:43:29:437.773 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:43:29:437.791 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:43:29:437.934 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:43:29:445.291 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.249:54246::10.0.0.141:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2026-03-30 Mon 03:43:29:445.333 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.249:54246::10.0.0.141:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:43:29:445.359 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:54246::10.0.0.141:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-30 Mon 03:43:29:445.371 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:54246::10.0.0.141:8086(-1):EvTcpClose 2026-03-30 Mon 03:43:29:445.636 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvTcpClose in state Established 2026-03-30 Mon 03:43:29:445.652 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Established : EvTcpClose 2026-03-30 Mon 03:43:29:446.032 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Connect : Start Connect timer 10.0.0.249:8086 2026-03-30 Mon 03:43:29:446.206 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:43:29:446.486 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842209445757 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = cn-jenkins-deploy-platform-ansible-os-5563-3. collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 334417 calls = 37 average_bytes = 9038 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 26 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 106, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 38, ] ] ] 2026-03-30 Mon 03:43:29:446.517 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:43:29:446.549 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:43:29:446.582 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842209445810 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.241:6379::Down.10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:43:29:446.592 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:43:29:446.604 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2026-03-30 Mon 03:43:29:446.617 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:43:29:446.858 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842209446175 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 334417 calls = 37 average_bytes = 9038 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 26 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 106, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 38, ] ] ] 2026-03-30 Mon 03:43:29:446.897 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:43:29:449.954 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:48354::10.0.0.249:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:43:29:449.983 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:48354::10.0.0.249:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:43:29:449.994 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:48354::10.0.0.249:8086(-1):EvTcpConnected 2026-03-30 Mon 03:43:29:451.641 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:43:29:451.672 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:43:29:451.718 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: ClientInit 2026-03-30 Mon 03:43:29:451.739 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-1.:contrail-query-engine:0:Database count 2 2026-03-30 Mon 03:43:29:451.850 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:43:29:451.877 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:43:29:451.916 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842209451708 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.241:6379::Down.10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:43:29:451.944 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:43:29:451.962 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:43:29:451.977 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:43:29:451.997 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:43:29:452.008 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:43:29:452.322 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842209451836 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 334417 calls = 37 average_bytes = 9038 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 26 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 107, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 38, ] ] ] 2026-03-30 Mon 03:43:29:452.348 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:43:29:461.836 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: OnMessage control in state: ClientInit 2026-03-30 Mon 03:43:29:461.917 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-30 Mon 03:43:29:461.928 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-30 Mon 03:43:29:461.952 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Received Ctrl Message with size 0 2026-03-30 Mon 03:43:29:462.679 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 334417 calls = 37 average_bytes = 9038 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 56 dequeues = 56 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 64 dequeues = 64 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Established state_since = 1774841849409167 last_state = Established last_event = scm::EvSandeshSend last_event_at = 1774842199435492 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 36 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 311791 calls = 35 average_bytes = 8908 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 99 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 26 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 107, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 38, ] ] max_sm_queue_count = 7 proxy seq 38 2026-03-30 Mon 03:43:29:462.712 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:43:29:462.737 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.241:6379::Down.10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 17 2026-03-30 Mon 03:43:29:462.749 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:43:29:462.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 6 2026-03-30 Mon 03:43:29:462.789 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:43:29:462.796 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:43:29:462.828 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Established 2026-03-30 Mon 03:43:34:099.150 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:43:34:099.346 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:43:34:099.659 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c26d70 2026-03-30 Mon 03:43:34:099.684 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:43:34:099.729 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:43:34:099.781 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c28930 2026-03-30 Mon 03:43:34:099.791 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:43:34:099.822 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:43:34:099.861 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:43:34:099.983 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:43:34:100.039 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:43:34:100.130 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:43:34:100.150 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:43:34:100.189 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:43:34:100.601 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c276b0 2026-03-30 Mon 03:43:34:100.623 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:43:34:100.643 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:43:34:100.661 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c29270 2026-03-30 Mon 03:43:34:100.669 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:43:34:100.683 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:43:34:100.696 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c27ff0 2026-03-30 Mon 03:43:34:100.704 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:43:34:100.717 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:43:34:100.745 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:43:34:100.760 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:43:34:100.886 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:43:34:101.769 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:43:34:101.805 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:43:34:101.830 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:43:34:101.843 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:43:34:101.857 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:43:34:101.884 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:43:34:101.912 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:43:34:101.930 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:43:34:101.946 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:43:34:437.825 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:43:34:438.018 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:43:34:438.077 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:43:34:438.118 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:43:34:438.154 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:43:34:438.391 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:43:34:438.433 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:43:34:438.446 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:43:34:438.457 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:43:34:438.471 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:43:39:438.466 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:43:39:438.580 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:43:39:438.613 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:43:39:438.635 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:43:39:438.671 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:43:39:438.965 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2ae30 2026-03-30 Mon 03:43:39:438.983 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:43:39:439.005 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:43:39:439.032 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c29bb0 2026-03-30 Mon 03:43:39:439.038 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:43:39:439.048 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:43:39:439.056 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2a4f0 2026-03-30 Mon 03:43:39:439.061 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:43:39:439.068 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:43:39:439.075 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2c0b0 2026-03-30 Mon 03:43:39:439.080 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:43:39:439.087 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:43:39:439.094 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2b770 2026-03-30 Mon 03:43:39:439.098 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:43:39:439.105 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:43:39:439.380 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:43:39:439.397 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:43:39:439.423 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:43:39:439.436 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:43:39:439.445 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:43:39:439.468 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:43:39:439.477 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:43:39:441.437 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:43:39:441.486 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:43:39:441.514 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:43:39:441.527 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:43:39:441.547 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:43:39:441.562 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:43:39:441.626 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:43:39:441.649 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:43:58:405.594 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-30 Mon 03:44:58:408.183 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-30 Mon 03:45:58:416.721 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-03-30 Mon 03:46:32:570.707 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c29270 2026-03-30 Mon 03:46:32:570.779 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:570.929 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c27ff0 2026-03-30 Mon 03:46:32:570.949 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:570.999 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c276b0 2026-03-30 Mon 03:46:32:571.014 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:571.057 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-30 Mon 03:46:32:571.067 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c26d70 2026-03-30 Mon 03:46:32:571.080 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:571.106 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c28930 2026-03-30 Mon 03:46:32:571.120 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:574.981 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:46:32:575.109 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:46:32:575.134 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:46:32:575.152 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:46:32:575.170 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:46:32:694.625 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c26470 2026-03-30 Mon 03:46:32:694.870 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:695.056 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c25b50 2026-03-30 Mon 03:46:32:695.081 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:695.114 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c25400 2026-03-30 Mon 03:46:32:695.127 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:695.158 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c20da0 2026-03-30 Mon 03:46:32:695.172 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:695.208 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-30 Mon 03:46:32:695.219 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c24830 2026-03-30 Mon 03:46:32:695.231 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:695.548 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:46:32:695.603 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:46:32:695.625 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:46:32:695.781 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:46:32:695.805 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:46:32:741.457 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c2c0b0 2026-03-30 Mon 03:46:32:741.503 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:741.541 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c2b770 2026-03-30 Mon 03:46:32:741.551 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:741.570 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c2a4f0 2026-03-30 Mon 03:46:32:741.579 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:741.744 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-30 Mon 03:46:32:741.754 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c29bb0 2026-03-30 Mon 03:46:32:741.763 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:741.792 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Disconnect status: -1 0x1c2ae30 2026-03-30 Mon 03:46:32:741.802 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-30 Mon 03:46:32:747.590 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:46:32:747.627 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:46:32:747.641 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:46:32:747.685 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:46:32:747.705 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:46:32:799.614 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.249:48354::10.0.0.249:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:46:32:799.677 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:48354::10.0.0.249:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-30 Mon 03:46:32:799.691 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:48354::10.0.0.249:8086(-1):EvTcpClose 2026-03-30 Mon 03:46:32:801.602 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvTcpClose in state Established 2026-03-30 Mon 03:46:32:801.631 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Established : EvTcpClose 2026-03-30 Mon 03:46:32:801.892 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Connect : Start Connect timer 10.0.0.241:8086 2026-03-30 Mon 03:46:32:801.988 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:32:802.159 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: SANDESH: Send FAILED: 1774842392801712 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = cn-jenkins-deploy-platform-ansible-os-5563-1. collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 545151 calls = 61 average_bytes = 8936 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 166, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 65, ] ] ] 2026-03-30 Mon 03:46:32:802.178 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:802.189 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:32:802.217 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: SANDESH: Send FAILED: 1774842392801743 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:32:802.230 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:32:802.242 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2026-03-30 Mon 03:46:32:802.254 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:32:802.435 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: SANDESH: Send FAILED: 1774842392801975 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 545151 calls = 61 average_bytes = 8936 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 166, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 65, ] ] ] 2026-03-30 Mon 03:46:32:802.452 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:805.354 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:38446::10.0.0.241:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:46:32:805.387 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:38446::10.0.0.241:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:46:32:805.397 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:38446::10.0.0.241:8086(-1):EvTcpConnected 2026-03-30 Mon 03:46:32:818.405 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:46:32:818.441 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:46:32:818.496 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: ClientInit 2026-03-30 Mon 03:46:32:818.512 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-1.:contrail-query-engine:0:Database count 3 2026-03-30 Mon 03:46:32:818.777 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:32:818.794 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:46:32:818.828 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842392818485 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:32:818.839 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:46:32:818.851 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:32:818.861 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:46:32:818.877 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:32:818.886 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:819.141 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842392818757 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 545151 calls = 61 average_bytes = 8936 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 167, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 65, ] ] ] 2026-03-30 Mon 03:46:32:819.170 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:827.209 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:38446::10.0.0.241:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-30 Mon 03:46:32:827.293 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:38446::10.0.0.241:8086(-1):EvTcpClose 2026-03-30 Mon 03:46:32:834.743 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-30 Mon 03:46:32:834.771 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: ClientInit : EvTcpClose 2026-03-30 Mon 03:46:32:834.905 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Return to idle with 1000 2026-03-30 Mon 03:46:32:834.914 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Idle 2026-03-30 Mon 03:46:32:834.985 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:32:835.248 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SANDESH: Send FAILED: 1774842392834853 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 546724 calls = 62 average_bytes = 8818 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 167, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 66, ] ] ] 2026-03-30 Mon 03:46:32:835.278 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:32:835.290 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:32:835.313 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SANDESH: Send FAILED: 1774842392834884 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:32:835.320 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:32:835.328 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:46:32:835.337 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:32:835.463 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SANDESH: Send FAILED: 1774842392834976 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 546724 calls = 62 average_bytes = 8818 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 167, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 66, ] ] ] 2026-03-30 Mon 03:46:32:835.477 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:33:418.153 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:33:418.319 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842393418106 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-30 Mon 03:46:33:418.341 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-03-30 Mon 03:46:33:835.537 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:46:33:835.835 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Connect : Start Connect timer 10.0.0.141:8086 2026-03-30 Mon 03:46:33:836.053 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:33:836.107 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842393835641 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:33:836.142 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:33:836.160 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:33:836.533 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842393836028 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 546724 calls = 62 average_bytes = 8818 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 167, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 34, _iter59->first = sent _iter59->second = 66, ] ] ] 2026-03-30 Mon 03:46:33:836.563 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:33:843.618 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:47746::10.0.0.141:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:46:33:843.659 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:47746::10.0.0.141:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:46:33:843.670 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:47746::10.0.0.141:8086(-1):EvTcpConnected 2026-03-30 Mon 03:46:33:846.670 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:46:33:846.735 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:46:33:846.810 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: ClientInit 2026-03-30 Mon 03:46:33:846.842 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-1.:contrail-query-engine:0:Database count 4 2026-03-30 Mon 03:46:33:847.036 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:33:847.056 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:46:33:847.105 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: SANDESH: Send FAILED: 1774842393846794 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:33:847.119 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:46:33:847.137 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:33:847.150 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:46:33:847.171 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:33:847.183 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:46:33:847.534 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: SANDESH: Send FAILED: 1774842393847014 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 546724 calls = 62 average_bytes = 8818 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 168, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 66, ] ] ] 2026-03-30 Mon 03:46:33:847.580 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706276755200, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:46:35:801.495 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:47746::10.0.0.141:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-30 Mon 03:46:35:825.042 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:47746::10.0.0.141:8086(-1):EvTcpClose 2026-03-30 Mon 03:46:35:825.416 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-30 Mon 03:46:35:825.434 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: ClientInit : EvTcpClose 2026-03-30 Mon 03:46:35:825.640 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Return to idle with 1000 2026-03-30 Mon 03:46:35:825.676 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Idle 2026-03-30 Mon 03:46:35:825.800 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:35:826.060 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842395825530 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 548297 calls = 63 average_bytes = 8703 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 168, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 67, ] ] ] 2026-03-30 Mon 03:46:35:826.093 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:35:826.108 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:35:826.139 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842395825575 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:35:826.149 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:35:826.162 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:46:35:826.191 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:35:826.407 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842395825771 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 548297 calls = 63 average_bytes = 8703 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 168, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 67, ] ] ] 2026-03-30 Mon 03:46:35:826.429 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:36:827.485 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:46:36:830.580 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Connect : Start Connect timer 10.0.0.249:8086 2026-03-30 Mon 03:46:36:830.808 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:36:830.881 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842396830343 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:36:830.900 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:36:830.920 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:36:831.275 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842396830769 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 548297 calls = 63 average_bytes = 8703 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 168, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 41, _iter59->first = sent _iter59->second = 67, ] ] ] 2026-03-30 Mon 03:46:36:831.340 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:36:835.940 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:40886::10.0.0.249:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:46:36:836.005 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:40886::10.0.0.249:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:46:36:836.072 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:40886::10.0.0.249:8086(-1):EvTcpConnected 2026-03-30 Mon 03:46:36:836.769 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:46:36:836.796 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:46:36:836.857 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: ClientInit 2026-03-30 Mon 03:46:36:836.886 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-1.:contrail-query-engine:0:Database count 5 2026-03-30 Mon 03:46:36:837.068 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:36:837.088 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:46:36:837.136 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842396836844 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:36:837.165 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:46:36:837.184 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:36:837.199 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:46:36:837.219 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:36:837.232 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:46:36:837.545 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842396837046 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 548297 calls = 63 average_bytes = 8703 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 67, ] ] ] 2026-03-30 Mon 03:46:36:837.574 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:46:37:575.118 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:46:37:575.293 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:46:37:575.343 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:46:37:575.370 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:46:37:575.400 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:46:37:575.749 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:46:37:575.784 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:46:37:575.800 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:46:37:575.814 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:46:37:575.826 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:46:37:695.816 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:46:37:696.541 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:46:37:696.638 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:46:37:696.706 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:46:37:696.729 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:46:37:696.852 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:46:37:696.963 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:46:37:697.047 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:46:37:697.077 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:46:37:697.102 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:46:37:747.722 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:46:37:747.929 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:46:37:748.047 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:46:37:748.174 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:46:37:748.306 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:46:37:748.481 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:46:37:748.565 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:46:37:748.606 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:46:37:748.663 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:46:37:748.700 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:46:40:837.791 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: 10.0.0.249:8086 EvConnectTimerExpired in state ClientInit 2026-03-30 Mon 03:46:40:837.866 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2026-03-30 Mon 03:46:40:837.878 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: ClientInit : EvConnectTimerExpired 2026-03-30 Mon 03:46:40:838.086 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Return to idle with 1000 2026-03-30 Mon 03:46:40:838.101 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Idle 2026-03-30 Mon 03:46:40:838.187 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:40:838.465 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842400837990 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 549870 calls = 64 average_bytes = 8591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 68, ] ] ] 2026-03-30 Mon 03:46:40:838.490 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:40:838.504 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:40:838.537 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842400838029 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:40:838.545 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:40:838.553 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:46:40:838.561 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:40:838.727 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842400838173 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 549870 calls = 64 average_bytes = 8591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 68, ] ] ] 2026-03-30 Mon 03:46:40:838.745 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:41:838.194 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:46:41:838.387 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Connect : Start Connect timer 10.0.0.241:8086 2026-03-30 Mon 03:46:41:838.494 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:41:838.549 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842401838264 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:41:838.563 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:41:838.576 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:41:838.857 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842401838479 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 549870 calls = 64 average_bytes = 8591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 68, ] ] ] 2026-03-30 Mon 03:46:41:838.899 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:41:841.735 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:47772::10.0.0.241:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:46:41:841.767 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:47772::10.0.0.241:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:46:41:841.777 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:47772::10.0.0.241:8086(-1):EvTcpConnected 2026-03-30 Mon 03:46:41:842.679 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:46:41:842.712 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:46:41:842.749 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: ClientInit 2026-03-30 Mon 03:46:41:842.766 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-1.:contrail-query-engine:0:Database count 6 2026-03-30 Mon 03:46:41:842.855 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:41:842.864 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:46:41:843.172 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: SANDESH: Send FAILED: 1774842401842742 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:41:843.180 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:46:41:843.193 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:41:843.200 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:46:41:843.211 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:46:41:843.217 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:46:41:843.372 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: SANDESH: Send FAILED: 1774842401842844 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 549870 calls = 64 average_bytes = 8591 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 170, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 68, ] ] ] 2026-03-30 Mon 03:46:41:843.392 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:46:42:575.817 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:46:42:575.949 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:46:42:575.997 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:46:42:576.030 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:46:42:576.058 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:46:42:576.296 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:46:42:576.325 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:46:42:576.339 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:46:42:576.350 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:46:42:576.361 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:46:42:696.730 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:46:42:696.877 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:46:42:696.903 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:46:42:696.940 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:46:42:697.086 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:46:42:697.164 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:46:42:697.180 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:46:42:697.208 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:46:42:697.239 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:46:42:697.252 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:46:42:748.557 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:46:42:748.702 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:46:42:748.745 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:46:42:748.773 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:46:42:748.806 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:46:42:748.955 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:46:42:749.018 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:46:42:749.032 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:46:42:749.043 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:46:42:749.053 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:46:44:470.905 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.249:47772::10.0.0.241:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2026-03-30 Mon 03:46:44:470.938 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.249:47772::10.0.0.241:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:46:44:470.957 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:47772::10.0.0.241:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-30 Mon 03:46:44:470.964 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:47772::10.0.0.241:8086(-1):EvTcpClose 2026-03-30 Mon 03:46:44:473.468 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-30 Mon 03:46:44:473.491 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: ClientInit : EvTcpClose 2026-03-30 Mon 03:46:44:473.629 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Return to idle with 1000 2026-03-30 Mon 03:46:44:473.638 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Idle 2026-03-30 Mon 03:46:44:473.696 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:44:473.839 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SANDESH: Send FAILED: 1774842404473575 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 172, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:46:44:473.856 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:44:473.866 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:44:473.887 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SANDESH: Send FAILED: 1774842404473605 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:44:473.897 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:44:473.905 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:46:44:473.915 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:44:474.050 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SANDESH: Send FAILED: 1774842404473686 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 172, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:46:44:474.063 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:45:478.484 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:46:45:478.676 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Connect : Start Connect timer 10.0.0.141:8086 2026-03-30 Mon 03:46:45:478.778 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:45:478.806 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: SANDESH: Send FAILED: 1774842405478542 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:45:478.814 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:45:478.822 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:46:45:478.957 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: SANDESH: Send FAILED: 1774842405478761 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 172, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 55, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:46:45:478.971 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707055699712, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:45:488.011 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:46:45:488.033 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:46:45:490.379 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:46:45:490.396 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:46:45:490.537 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:46:45:490.574 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Idle 2026-03-30 Mon 03:46:45:490.633 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:45:490.774 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SANDESH: Send FAILED: 1774842405490484 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 172, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:46:45:490.792 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:45:490.802 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:45:490.826 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SANDESH: Send FAILED: 1774842405490513 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:46:45:490.833 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:46:45:490.842 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:46:45:490.850 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:45:490.975 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SANDESH: Send FAILED: 1774842405490622 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 173, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:46:45:490.987 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:47:576.406 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:46:47:576.553 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:46:47:576.606 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:46:47:576.654 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:46:47:576.692 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:46:47:576.893 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:46:47:576.934 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:46:47:576.953 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:46:47:576.972 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:46:47:576.991 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:46:47:696.947 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:46:47:697.123 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:46:47:697.154 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:46:47:697.224 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:46:47:697.242 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:46:47:697.299 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:46:47:697.315 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:46:47:697.352 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:46:47:697.398 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:46:47:697.417 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:46:47:749.047 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:46:47:749.188 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:46:47:749.228 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:46:47:749.263 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:46:47:749.291 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:46:47:749.466 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:46:47:749.502 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:46:47:749.523 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:46:47:749.536 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:46:47:749.548 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:46:52:576.987 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:46:52:577.158 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:46:52:577.216 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:46:52:577.264 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:46:52:577.301 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:46:52:577.449 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:46:52:579.408 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:46:52:579.434 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:46:52:579.450 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:46:52:579.466 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:46:52:697.199 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:46:52:699.993 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:46:52:701.679 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:46:52:701.760 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:46:52:701.792 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:46:52:701.819 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:46:52:701.854 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:46:52:701.870 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:46:52:701.881 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:46:52:701.891 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:46:52:749.582 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:46:52:751.607 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:46:52:751.695 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:46:52:751.755 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:46:52:751.801 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:46:52:751.861 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:46:52:751.901 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:46:52:751.925 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:46:52:751.979 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:46:52:752.002 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:46:57:579.480 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:46:57:579.786 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:46:57:579.886 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:46:57:579.955 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:46:57:580.046 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:46:57:580.339 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:46:57:580.453 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:46:57:580.494 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:46:57:580.528 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:46:57:580.582 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:46:57:701.770 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:46:57:702.082 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:46:57:702.142 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:46:57:702.231 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:46:57:702.303 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:46:57:702.370 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:46:57:702.477 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:46:57:702.520 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:46:57:702.553 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:46:57:702.585 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:46:57:751.961 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:46:57:752.105 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:46:57:752.144 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:46:57:752.168 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:46:57:752.194 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:46:57:752.510 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:46:57:752.539 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:46:57:752.554 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:46:57:752.565 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:46:57:752.577 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:46:58:419.547 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2026-03-30 Mon 03:46:58:419.611 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:58:419.711 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SANDESH: Send FAILED: 1774842418419391 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 129 dequeues = 129 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 161 dequeues = 161 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842405490573 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842405490849 ] max_sm_queue_count = 7 ] 2026-03-30 Mon 03:46:58:419.741 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:46:58:419.755 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:46:58:419.911 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: SANDESH: Send FAILED: 1774842418419524 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 173, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:46:58:419.930 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706285152000, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:01:490.774 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:47:01:491.077 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Connect : Start Connect timer 10.0.0.249:8086 2026-03-30 Mon 03:47:01:491.106 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:47:01:491.160 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:47:01:491.177 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:47:01:491.206 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774842421490858 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:47:01:491.224 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:47:01:491.233 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:47:01:491.240 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:47:01:491.328 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:47:01:491.357 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Idle 2026-03-30 Mon 03:47:01:491.417 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:01:491.670 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774842421491165 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 173, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 62, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:47:01:491.707 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:01:491.722 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:01:491.857 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774842421491288 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 173, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:47:01:491.871 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:01:491.879 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:01:491.901 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774842421491308 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:47:01:491.909 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:47:01:491.918 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:47:01:491.936 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:01:492.149 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774842421491406 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 174, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:47:01:492.171 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:02:580.492 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:02:580.775 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:02:580.866 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:02:580.945 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:02:581.030 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:02:581.520 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:02:581.591 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:02:581.616 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:02:581.629 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:02:581.641 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:02:702.228 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:02:702.481 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:02:702.565 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:02:702.707 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:02:702.820 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:02:702.869 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:02:702.927 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:02:703.010 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:02:703.119 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:02:703.166 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:02:752.652 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:02:752.944 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:02:753.113 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:02:753.233 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:02:753.390 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:02:753.531 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:02:753.617 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:02:753.674 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:02:753.730 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:02:753.782 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:07:581.667 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:07:581.991 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:07:582.138 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:07:582.230 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:07:582.312 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:07:582.573 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:07:582.649 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:07:582.689 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:07:582.718 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:07:582.745 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:07:702.720 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:07:703.187 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:07:703.275 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:07:703.426 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:07:703.603 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:07:703.684 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:07:703.733 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:07:703.836 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:07:703.943 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:07:703.991 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:07:753.616 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:07:753.804 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:07:753.873 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:07:753.916 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:07:753.961 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:07:754.255 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:07:754.302 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:07:754.323 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:07:754.341 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:07:754.359 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:12:582.657 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:12:582.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:12:582.819 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:12:582.849 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:12:582.879 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:12:583.140 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:12:583.177 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:12:583.193 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:12:583.207 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:12:583.219 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:12:703.300 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:12:703.453 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:12:703.668 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:12:703.740 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:12:703.758 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:12:703.805 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:12:703.990 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:12:704.049 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:12:704.076 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:12:704.117 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:12:754.354 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:12:754.518 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:12:754.589 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:12:754.634 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:12:754.678 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:12:754.867 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:12:754.910 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:12:754.933 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:12:754.951 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:12:754.969 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:17:583.244 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:17:583.453 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:17:583.530 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:17:583.589 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:17:583.639 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:17:583.879 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:17:583.930 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:17:583.953 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:17:583.976 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:17:583.996 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:17:703.566 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:17:703.703 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:17:703.790 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:17:703.857 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:17:703.873 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:17:703.907 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:17:704.099 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:17:704.168 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:17:704.182 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:17:704.216 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:17:754.976 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:17:755.243 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:17:755.329 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:17:755.368 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:17:755.400 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:17:755.544 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:17:755.585 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:17:755.600 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:17:755.613 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:17:755.628 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:22:583.974 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:22:584.153 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:22:585.294 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:22:585.348 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:22:585.386 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:22:585.444 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:22:585.593 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:22:585.621 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:22:585.638 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:22:585.654 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:22:703.784 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:22:703.934 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:22:703.957 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:22:704.012 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:22:704.026 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:22:704.059 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:22:704.195 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:22:704.248 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:22:704.260 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:22:704.291 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:22:755.633 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:22:755.762 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:22:755.815 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:22:755.848 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:22:755.897 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:22:756.116 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:22:756.187 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:22:756.202 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:22:756.215 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:22:756.228 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:27:585.542 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:27:585.740 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:27:585.804 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:27:585.853 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:27:585.897 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:27:586.044 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:27:586.084 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:27:586.141 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:27:586.176 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:27:586.193 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:27:704.452 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:27:704.590 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:27:704.635 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:27:704.677 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:27:704.721 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:27:704.781 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:27:704.816 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:27:704.837 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:27:704.856 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:27:704.877 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:27:756.191 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:27:756.315 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:27:756.355 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:27:756.404 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:27:756.434 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:27:756.539 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:27:756.618 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:27:756.634 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:27:756.647 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:27:756.659 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:28:422.779 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:28:422.913 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: SANDESH: Send FAILED: 1774842448422582 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 136 dequeues = 136 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 171 dequeues = 171 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842421491355 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842421491934 ] max_sm_queue_count = 7 ] 2026-03-30 Mon 03:47:28:422.940 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:28:422.950 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:28:423.090 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: SANDESH: Send FAILED: 1774842448422723 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 174, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 67, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:47:28:423.115 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:31:491.791 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707085743872, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:47:31:492.029 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707085743872, Pid 1]: Connect : Start Connect timer 10.0.0.241:8086 2026-03-30 Mon 03:47:31:492.186 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707085743872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:47:31:492.230 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707085743872, Pid 1]: SANDESH: Send FAILED: 1774842451491877 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.241:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:47:31:492.242 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707085743872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:47:31:492.255 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707085743872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:47:31:492.537 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707085743872, Pid 1]: SANDESH: Send FAILED: 1774842451492165 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.241:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 174, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 69, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:47:31:492.569 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707085743872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:31:495.339 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:47:31:495.363 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:47:31:498.267 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:47:31:498.292 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:47:31:498.457 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:47:31:498.496 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Idle 2026-03-30 Mon 03:47:31:498.578 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:31:498.831 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774842451498392 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 174, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:47:31:498.858 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:31:498.872 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:31:498.905 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774842451498426 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:47:31:498.914 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:47:31:498.926 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:47:31:498.939 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:31:499.183 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: SANDESH: Send FAILED: 1774842451498566 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 175, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:47:31:499.202 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706268358400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:32:586.136 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:32:586.295 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:32:586.342 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:32:586.385 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:32:586.427 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:32:586.742 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:32:586.775 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:32:586.790 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:32:586.803 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:32:586.815 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:32:704.850 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:32:705.007 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:32:705.042 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:32:705.081 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:32:705.144 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:32:705.182 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:32:705.233 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:32:705.257 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:32:705.275 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:32:705.292 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:32:756.638 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:32:756.791 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:32:756.850 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:32:756.878 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:32:756.904 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:32:757.029 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:32:757.069 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:32:757.083 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:32:757.108 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:32:757.122 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:33:421.344 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707064092416, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:33:421.427 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707064092416, Pid 1]: SANDESH: Send FAILED: 1774842453421298 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-30 Mon 03:47:33:421.439 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707064092416, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-03-30 Mon 03:47:37:586.874 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:37:587.077 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:37:587.176 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:37:587.240 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:37:587.304 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:37:587.460 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:37:587.509 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:37:587.537 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:37:587.565 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:37:587.745 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:37:705.147 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:37:705.473 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:37:705.527 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:37:705.612 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:37:705.679 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:37:705.742 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:37:705.820 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:37:705.858 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:37:705.888 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:37:705.917 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:37:757.111 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:37:757.242 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:37:757.290 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:37:757.321 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:37:757.352 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:37:757.545 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:37:757.582 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:37:757.597 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:37:757.608 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:37:757.662 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:42:587.552 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:42:587.732 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:42:587.801 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:42:587.835 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:42:587.868 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:42:588.004 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:42:588.034 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:42:588.057 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:42:588.070 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:42:588.082 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:42:705.559 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:42:705.684 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:42:705.862 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:42:705.925 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:42:705.947 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:42:705.974 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:42:706.009 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:42:706.022 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:42:706.031 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:42:706.061 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:42:757.615 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:42:757.742 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:42:757.777 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:42:757.805 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:42:757.832 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:42:757.997 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:42:760.491 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:42:760.539 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:42:760.554 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:42:760.567 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:47:588.496 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:47:588.629 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:47:588.674 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:47:588.719 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:47:588.769 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:47:590.003 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:47:590.046 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:47:590.061 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:47:590.089 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:47:590.116 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:47:705.778 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:47:705.910 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:47:706.732 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:47:706.817 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:47:706.862 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:47:706.899 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:47:706.944 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:47:706.968 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:47:706.984 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:47:707.000 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:47:758.193 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:47:759.298 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:47:760.564 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:47:760.657 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:47:760.700 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:47:760.737 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:47:760.891 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:47:760.926 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:47:760.967 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:47:760.983 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:52:590.092 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:52:590.240 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:52:590.296 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:52:590.335 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:52:590.375 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:52:590.554 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:52:590.584 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:52:590.597 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:52:590.772 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:52:590.788 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:52:705.980 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:52:706.146 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:52:707.008 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:52:707.114 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:52:707.172 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:52:707.196 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:52:707.208 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:52:707.252 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:52:707.308 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:52:707.324 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:52:759.445 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:52:759.767 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:52:760.936 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:52:761.295 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:52:761.356 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:52:761.392 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:52:761.559 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:52:761.619 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:52:761.641 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:52:761.659 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:57:590.641 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:47:57:590.812 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:47:57:590.856 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:47:57:590.900 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:47:57:590.936 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:47:57:591.249 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c28930 2026-03-30 Mon 03:47:57:591.300 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c27ff0 2026-03-30 Mon 03:47:57:591.319 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29270 2026-03-30 Mon 03:47:57:591.337 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26d70 2026-03-30 Mon 03:47:57:591.356 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c276b0 2026-03-30 Mon 03:47:57:706.223 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:47:57:706.366 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c26470 2026-03-30 Mon 03:47:57:707.226 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:47:57:707.314 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:47:57:707.368 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25b50 2026-03-30 Mon 03:47:57:707.388 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c25400 2026-03-30 Mon 03:47:57:707.397 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:47:57:707.425 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:47:57:707.453 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c20da0 2026-03-30 Mon 03:47:57:707.466 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c24830 2026-03-30 Mon 03:47:57:760.535 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:47:57:760.915 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2b770 2026-03-30 Mon 03:47:57:761.626 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:47:57:761.726 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:47:57:761.768 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:47:57:761.804 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:47:57:762.018 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c29bb0 2026-03-30 Mon 03:47:57:762.060 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2a4f0 2026-03-30 Mon 03:47:57:762.076 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2c0b0 2026-03-30 Mon 03:47:57:770.690 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: -1 0x1c2ae30 2026-03-30 Mon 03:47:58:426.514 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706289350400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2026-03-30 Mon 03:47:58:426.585 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706289350400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:58:426.700 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706289350400, Pid 1]: SANDESH: Send FAILED: 1774842478426346 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 144 dequeues = 144 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 182 dequeues = 182 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842451498494 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842453421324 ] max_sm_queue_count = 7 ] 2026-03-30 Mon 03:47:58:426.718 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706289350400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:47:58:426.732 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706289350400, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:47:58:426.992 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706289350400, Pid 1]: SANDESH: Send FAILED: 1774842478426492 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 175, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 75, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:47:58:427.027 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706289350400, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:01:499.190 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:48:01:499.363 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Connect : Start Connect timer 10.0.0.141:8086 2026-03-30 Mon 03:48:01:499.452 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:48:01:499.480 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842481499260 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.141:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:48:01:499.488 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:01:499.503 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:48:01:499.639 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842481499441 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.141:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 175, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 77, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:48:01:499.653 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:01:504.003 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-30 Mon 03:48:01:504.032 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: -(-1):EvTcpConnectFail 2026-03-30 Mon 03:48:01:507.687 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-30 Mon 03:48:01:507.718 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Connect : EvTcpConnectFail 2026-03-30 Mon 03:48:01:507.870 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-30 Mon 03:48:01:507.905 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Idle 2026-03-30 Mon 03:48:01:507.966 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:01:508.130 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: SANDESH: Send FAILED: 1774842481507801 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 175, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 79, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:48:01:508.147 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:01:508.157 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:01:508.179 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: SANDESH: Send FAILED: 1774842481507840 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:48:01:508.187 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:01:508.199 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:48:01:508.216 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:01:508.347 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: SANDESH: Send FAILED: 1774842481507956 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 157 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 176, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 79, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:48:01:508.359 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706272556800, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:02:591.327 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c28930 2026-03-30 Mon 03:48:02:591.443 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c27ff0 2026-03-30 Mon 03:48:02:591.477 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29270 2026-03-30 Mon 03:48:02:591.499 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26d70 2026-03-30 Mon 03:48:02:591.525 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c276b0 2026-03-30 Mon 03:48:02:591.769 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c28930 2026-03-30 Mon 03:48:02:591.777 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:591.805 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:591.828 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c27ff0 2026-03-30 Mon 03:48:02:591.833 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:591.841 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:591.868 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c29270 2026-03-30 Mon 03:48:02:591.873 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:591.881 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:591.888 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c26d70 2026-03-30 Mon 03:48:02:591.922 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:591.930 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:591.971 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c276b0 2026-03-30 Mon 03:48:02:591.977 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:591.985 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:592.160 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:592.172 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:592.196 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:592.239 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:592.248 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:594.410 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:02:594.461 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842482592278 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.249:6379::Down.10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:02:594.480 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:02:594.503 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:594.525 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:594.535 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:594.556 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:594.569 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:594.577 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:594.588 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:594.596 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:594.612 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:594.627 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:708.471 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c26470 2026-03-30 Mon 03:48:02:708.613 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25b50 2026-03-30 Mon 03:48:02:708.674 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c25400 2026-03-30 Mon 03:48:02:708.727 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c20da0 2026-03-30 Mon 03:48:02:708.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c24830 2026-03-30 Mon 03:48:02:708.834 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c26470 2026-03-30 Mon 03:48:02:708.853 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:708.881 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:708.894 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c25b50 2026-03-30 Mon 03:48:02:708.901 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:708.914 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:708.925 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c25400 2026-03-30 Mon 03:48:02:708.933 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:708.945 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:708.955 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c20da0 2026-03-30 Mon 03:48:02:708.962 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:708.973 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:708.984 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c24830 2026-03-30 Mon 03:48:02:708.991 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:709.002 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:711.648 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:711.685 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:711.706 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:711.719 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:711.736 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:711.749 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:711.766 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:711.791 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:711.810 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:711.824 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:711.860 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:713.549 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:02:713.605 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842482711899 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:02:713.618 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:02:715.442 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:715.491 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:715.514 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:715.536 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:760.987 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2b770 2026-03-30 Mon 03:48:02:761.323 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2b770 2026-03-30 Mon 03:48:02:761.343 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:761.370 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:761.615 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:761.643 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:761.675 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:762.079 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c29bb0 2026-03-30 Mon 03:48:02:762.165 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2a4f0 2026-03-30 Mon 03:48:02:762.207 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2c0b0 2026-03-30 Mon 03:48:02:762.476 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c29bb0 2026-03-30 Mon 03:48:02:762.492 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:762.509 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:762.531 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2a4f0 2026-03-30 Mon 03:48:02:762.538 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:762.550 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:762.560 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2c0b0 2026-03-30 Mon 03:48:02:762.567 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:762.578 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:762.901 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:762.920 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:765.200 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:02:765.252 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: SANDESH: Send FAILED: 1774842482762961 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:02:765.273 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:02:767.577 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:767.623 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:767.639 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:767.660 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:767.674 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:767.699 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:767.720 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:02:771.843 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1c2ae30 2026-03-30 Mon 03:48:02:772.220 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: RAC_Connect status: 0 0x1c2ae30 2026-03-30 Mon 03:48:02:772.244 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:48:02:772.262 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-30 Mon 03:48:02:772.498 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-30 Mon 03:48:02:772.516 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-30 Mon 03:48:02:772.543 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-30 Mon 03:48:28:428.023 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:28:428.121 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: SANDESH: Send FAILED: 1774842508427894 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 154 dequeues = 154 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 195 dequeues = 195 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842481507904 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842482765194 ] max_sm_queue_count = 7 ] 2026-03-30 Mon 03:48:28:428.140 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:28:428.149 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:48:28:428.292 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: SANDESH: Send FAILED: 1774842508427996 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 217 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 236, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 85, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:48:28:428.304 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707102537472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:31:509.247 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-30 Mon 03:48:31:509.449 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Connect : Start Connect timer 10.0.0.249:8086 2026-03-30 Mon 03:48:31:509.546 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:48:31:509.592 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842511509318 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:31:509.601 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:48:31:509.610 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:48:31:509.803 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: SANDESH: Send FAILED: 1774842511509534 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 217 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 236, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 87, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:48:31:509.822 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707098339072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:48:31:514.215 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:41908::10.0.0.249:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-30 Mon 03:48:31:516.286 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:41908::10.0.0.249:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:48:31:516.305 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707250407552, Pid 1]: 10.0.0.249:41908::10.0.0.249:8086(-1):EvTcpConnected 2026-03-30 Mon 03:48:31:518.675 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:48:31:518.690 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:48:31:518.731 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: ClientInit 2026-03-30 Mon 03:48:31:518.743 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-1.:contrail-query-engine:0:Database count 7 2026-03-30 Mon 03:48:31:518.844 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:48:31:518.853 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:48:31:518.877 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: SANDESH: Send FAILED: 1774842511518723 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:48:31:518.885 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:48:31:518.894 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:48:31:518.900 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:48:31:518.911 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:48:31:518.916 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:48:31:519.054 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: SANDESH: Send FAILED: 1774842511518833 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 217 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 237, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 89, _iter59->first = sent _iter59->second = 69, ] ] ] 2026-03-30 Mon 03:48:31:519.073 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707089942272, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:48:31:563.212 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: OnMessage control in state: ClientInit 2026-03-30 Mon 03:48:31:563.342 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-30 Mon 03:48:31:563.353 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-30 Mon 03:48:31:566.719 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: Received Ctrl Message with size 0 2026-03-30 Mon 03:48:31:574.548 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 551443 calls = 65 average_bytes = 8483 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 154 dequeues = 154 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 195 dequeues = 195 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774842481507904 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774842482765194 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 22 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 179219 calls = 19 average_bytes = 9432 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 217 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 237, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 89, _iter59->first = sent _iter59->second = 69, ] ] max_sm_queue_count = 7 proxy seq 90 2026-03-30 Mon 03:48:31:574.619 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:48:31:574.646 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 49 2026-03-30 Mon 03:48:31:574.662 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:48:31:574.712 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 11 2026-03-30 Mon 03:48:31:574.722 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:48:31:574.729 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:48:31:574.780 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707106735872, Pid 1]: Established 2026-03-30 Mon 03:48:58:428.432 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706289350400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2026-03-30 Mon 03:49:58:429.621 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139707064092416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2026-03-30 Mon 03:50:58:431.421 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2026-03-30 Mon 03:51:43:965.881 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvStop in state Established 2026-03-30 Mon 03:51:43:965.969 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Idle 2026-03-30 Mon 03:51:43:966.063 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-30 Mon 03:51:43:966.074 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-30 Mon 03:51:43:966.253 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: SANDESH: Send FAILED: 1774842703966052 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774841798038074 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, (*_iter40) = 10.0.0.249:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 753321 calls = 85 average_bytes = 8862 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 16 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 217 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 54 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 237, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 91, _iter59->first = sent _iter59->second = 92, ] ] ] 2026-03-30 Mon 03:51:43:966.278 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 139706280953600, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:52:12:367.568 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: http-server-port 8091 2026-03-30 Mon 03:52:12:367.633 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Max-tasks 16 2026-03-30 Mon 03:52:12:367.639 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Max-slice 100 2026-03-30 Mon 03:52:12:367.644 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Collectors 10.0.0.249:8086 2026-03-30 Mon 03:52:12:367.648 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Collectors 10.0.0.241:8086 2026-03-30 Mon 03:52:12:367.652 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Collectors 10.0.0.141:8086 2026-03-30 Mon 03:52:12:367.674 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: ROLE : Generator 2026-03-30 Mon 03:52:12:367.679 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-30 Mon 03:52:12:367.683 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5563-1. 2026-03-30 Mon 03:52:12:367.688 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-30 Mon 03:52:12:367.692 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-30 Mon 03:52:12:367.696 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-30 Mon 03:52:12:367.701 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-30 Mon 03:52:12:380.457 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: HTTP Introspect Init 2026-03-30 Mon 03:52:12:380.579 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 138 2026-03-30 Mon 03:52:12:380.666 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Sandesh Http Server Port: 8091 2026-03-30 Mon 03:52:12:382.194 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-30 Mon 03:52:12:382.237 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-30 Mon 03:52:12:382.631 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Idle 2026-03-30 Mon 03:52:12:382.773 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: No Client: 1774842732382717 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842732382642 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:52:12:382.893 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-30 Mon 03:52:12:382.906 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-30 Mon 03:52:12:382.917 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Cassandra Servers: 10.0.0.249:9042 10.0.0.241:9042 10.0.0.141:9042 2026-03-30 Mon 03:52:12:391.433 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Processing scm::EvStart in state Idle 2026-03-30 Mon 03:52:12:398.962 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Disconnect 2026-03-30 Mon 03:52:12:399.067 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-30 Mon 03:52:12:399.437 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Connect : Start Connect timer 10.0.0.249:8086 2026-03-30 Mon 03:52:12:399.512 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:52:12:399.530 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: SANDESH: Send FAILED: 1774842732398882 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:52:12:399.545 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:52:12:399.553 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:52:12:399.614 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: SANDESH: Send FAILED: 1774842732399051 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842732382642 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:52:12:399.623 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:52:12:399.630 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:52:12:399.676 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: SANDESH: Send FAILED: 1774842732399097 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842732382642 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:52:12:399.686 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:52:12:399.693 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:52:12:399.704 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: SANDESH: Send FAILED: 1774842732399121 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:52:12:399.709 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:52:12:399.715 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:52:12:399.754 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: SANDESH: Send FAILED: 1774842732399498 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774842732382642 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:52:12:399.762 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-30 Mon 03:52:12:427.919 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-30 Mon 03:52:12:431.005 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xed5580 2026-03-30 Mon 03:52:12:431.028 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:431.040 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:431.057 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xeda540 2026-03-30 Mon 03:52:12:431.061 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:431.068 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:431.075 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xedaeb0 2026-03-30 Mon 03:52:12:431.079 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:431.085 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:431.471 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530369406528, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-30 Mon 03:52:12:431.509 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530369406528, Pid 1]: SANDESH: Send FAILED: 1774842732430910 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:52:12:431.517 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530369406528, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-30 Mon 03:52:12:431.875 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xedbb70 2026-03-30 Mon 03:52:12:431.880 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:431.889 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:431.901 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xedc830 2026-03-30 Mon 03:52:12:431.905 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:431.911 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.249 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:431.918 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xedd4f0 2026-03-30 Mon 03:52:12:431.922 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:431.928 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:431.936 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xede0b0 2026-03-30 Mon 03:52:12:431.940 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:431.946 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:431.959 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xedecf0 2026-03-30 Mon 03:52:12:431.963 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:431.969 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:431.977 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xedf930 2026-03-30 Mon 03:52:12:431.981 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:431.986 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:431.994 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xee0570 2026-03-30 Mon 03:52:12:431.998 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:432.004 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.241 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:432.011 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xee11b0 2026-03-30 Mon 03:52:12:432.015 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:432.021 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:432.028 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xee1df0 2026-03-30 Mon 03:52:12:432.032 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:432.038 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:432.047 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xee2a30 2026-03-30 Mon 03:52:12:432.053 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:432.062 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:432.074 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xee3670 2026-03-30 Mon 03:52:12:432.080 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:432.088 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:432.097 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: RAC_Connect status: 0 0xee42b0 2026-03-30 Mon 03:52:12:432.101 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: Connected to REDIS... 2026-03-30 Mon 03:52:12:432.110 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.141 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-30 Mon 03:52:12:434.020 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:36432::10.0.0.249:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2026-03-30 Mon 03:52:12:436.058 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: 10.0.0.249:36432::10.0.0.249:8086(-1) OnSessionEvent TCP Connected 2026-03-30 Mon 03:52:12:436.073 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: 10.0.0.249:36432::10.0.0.249:8086(-1):EvTcpConnected 2026-03-30 Mon 03:52:12:436.355 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:436.452 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-30 Mon 03:52:12:436.467 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-30 Mon 03:52:12:436.501 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: ClientInit 2026-03-30 Mon 03:52:12:436.515 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5563-1.:contrail-query-engine:0:Database count 1 2026-03-30 Mon 03:52:12:436.589 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:52:12:436.597 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:52:12:436.613 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: SANDESH: Send FAILED: 1774842732436495 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-30 Mon 03:52:12:436.618 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:52:12:436.625 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:52:12:436.630 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-30 Mon 03:52:12:436.648 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:52:12:436.653 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-30 Mon 03:52:12:436.741 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: SANDESH: Send FAILED: 1774842732436577 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774842732382642 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 19, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-30 Mon 03:52:12:436.761 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-30 Mon 03:52:12:440.191 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.231 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.240 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.251 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.257 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.268 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.275 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.285 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.292 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.249:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.720 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:440.753 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:440.765 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:440.775 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:440.785 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:440.824 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.839 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.850 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.857 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.867 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.874 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.883 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.890 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.899 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.906 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.241:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.915 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.922 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.932 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.938 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.947 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.954 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.963 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.969 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:440.981 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-30 Mon 03:52:12:440.988 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.141:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-30 Mon 03:52:12:441.006 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:441.028 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:441.038 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:441.048 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:441.057 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:441.071 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:441.086 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:441.095 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:441.104 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:441.113 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-30 Mon 03:52:12:441.186 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:52:12:441.198 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:52:12:441.219 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: SANDESH: Send FAILED: 1774842732440700 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-30 Mon 03:52:12:441.225 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:52:12:441.236 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:52:12:441.242 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:52:12:441.257 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: SANDESH: Send FAILED: 1774842732440747 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.241:6379::Down.10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:52:12:441.262 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:52:12:441.268 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:52:12:441.273 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:52:12:441.284 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: SANDESH: Send FAILED: 1774842732441023 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = 10.0.0.141:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:52:12:441.288 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:52:12:441.294 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-30 Mon 03:52:12:441.299 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-30 Mon 03:52:12:441.310 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: SANDESH: Send FAILED: 1774842732441082 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-30 Mon 03:52:12:441.318 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-30 Mon 03:52:12:441.910 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: OnMessage control in state: ClientInit 2026-03-30 Mon 03:52:12:441.935 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-30 Mon 03:52:12:441.942 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-30 Mon 03:52:12:441.957 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: Received Ctrl Message with size 0 2026-03-30 Mon 03:52:12:442.034 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774842732382642 collector_name = collector_ip = 10.0.0.249:8086 collector_list= [ [ (*_iter40) = 10.0.0.249:8086, (*_iter40) = 10.0.0.241:8086, (*_iter40) = 10.0.0.141:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 19, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2026-03-30 Mon 03:52:12:442.055 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:52:12:442.074 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5563-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.249:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.249:9042, (*_iter6) = 10.0.0.241:9042, (*_iter6) = 10.0.0.141:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.249:6379, (*_iter6) = 10.0.0.241:6379, (*_iter6) = 10.0.0.141:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2026-03-30 Mon 03:52:12:442.082 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-30 Mon 03:52:12:442.088 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:52:12:442.094 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-30 Mon 03:52:12:442.117 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: Established 2026-03-30 Mon 03:53:12:383.791 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-30 Mon 03:54:12:385.387 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-30 Mon 03:55:12:386.156 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530330400320, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-30 Mon 03:56:12:386.835 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530463450688, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-30 Mon 03:57:12:387.758 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-30 Mon 03:58:12:388.276 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-30 Mon 03:59:12:388.994 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530330400320, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-30 Mon 04:00:12:392.092 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-30 Mon 04:01:12:393.701 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530377803328, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-03-30 Mon 04:02:12:395.008 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530463450688, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2026-03-30 Mon 04:03:12:396.410 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530377803328, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2026-03-30 Mon 04:04:12:396.896 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530463450688, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2026-03-30 Mon 04:05:12:398.321 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2026-03-30 Mon 04:06:12:399.274 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2026-03-30 Mon 04:07:12:401.131 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2026-03-30 Mon 04:08:12:402.628 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2026-03-30 Mon 04:09:12:403.540 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530471847488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2026-03-30 Mon 04:10:12:404.511 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530382001728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2026-03-30 Mon 04:11:12:407.809 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530382001728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2026-03-30 Mon 04:11:40:652.686 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:8091::10.0.0.249:57546 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-30 Mon 04:11:49:668.813 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:8091::10.0.0.249:58034 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-30 Mon 04:12:12:408.839 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530373604928, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2026-03-30 Mon 04:13:12:412.464 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530373604928, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2026-03-30 Mon 04:13:33:213.781 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:8091::10.0.0.249:49192 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-30 Mon 04:13:56:601.491 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.249 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-03-30 Mon 04:13:56:601.839 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID ded626e8-2bee-11f1-9bc4-00000a0000f9 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-03-30 Mon 04:13:56:601.935 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 :0:json query is: { end_time: 1774844036522051 , enqueue_time: 1774844036601023 , query_metadata: {"enqueue_time": 1774844036600898} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:13:56:602.150 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-03-30 Mon 04:13:56:602.172 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-03-30 Mon 04:13:56:602.249 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for ded626e8-2bee-11f1-9bc4-00000a0000f9 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-03-30 Mon 04:13:56:602.328 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530463450688, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 :0:json query is: { end_time: 1774844036522051 , enqueue_time: 1774844036601023 , query_metadata: {"enqueue_time": 1774844036600898} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:13:56:742.779 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:742.839 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:742.851 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:742.860 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:742.870 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:742.879 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:742.887 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:742.910 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:742.919 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:742.928 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:743.082 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:743.106 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:743.113 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:743.119 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:743.125 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:743.131 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:743.144 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:743.150 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:743.156 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 0 and 34 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:743.285 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 34 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:743.362 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 :0:json query is: { end_time: 1774844036522051 , enqueue_time: 1774844036601023 , query_metadata: {"enqueue_time": 1774844036600898} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:13:56:743.547 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:743.661 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:743.825 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:743.989 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:744.127 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:744.270 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:744.516 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:744.640 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:744.800 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:744.905 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:744.998 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:745.107 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:745.156 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 10 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:745.202 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:745.310 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:745.401 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:745.513 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:745.608 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:745.702 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:745.789 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:745.882 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:745.973 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:746.065 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:746.152 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:746.255 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:746.345 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:746.453 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:746.557 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:746.646 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:746.734 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:746.828 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:746.920 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:747.008 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:747.094 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:747.191 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:745.206 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:747.563 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 10 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:747.593 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:747.606 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 10 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:747.617 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:747.624 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 10 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:747.643 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:747.650 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 UNION between tables of sizes 10 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:56:747.660 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: [SYS_INFO]: QELog: ded626e8-2bee-11f1-9bc4-00000a0000f9 Resulting size of set 10 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:56:747.981 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:748.161 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:748.331 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:748.527 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:748.654 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:748.762 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:748.888 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:748.989 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:749.081 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:749.176 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QEQueryLog: ded626e8-2bee-11f1-9bc4-00000a0000f9 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:56:749.413 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-03-30 Mon 04:13:56:749.775 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-03-30 Mon 04:13:56:749.980 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":6, "count":44} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-03-30 Mon 04:13:56:750.520 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530373604928, Pid 1]: [SYS_INFO]: QELog: Finished: QID ded626e8-2bee-11f1-9bc4-00000a0000f9 Table ObjectVNTable Time(ms) 148 RedisTime(ms) 0 MergeTime(ms) 0 Rows 44 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-03-30 Mon 04:13:56:750.582 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530476045888, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5563-1. table = ObjectVNTable query_stats= [ qid = ded626e8-2bee-11f1-9bc4-00000a0000f9 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-25949624:ctest-vn22-57996482", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 140,0, 140,0, 145,0, chunk_select_time = 0,0, 0,3, 0,1, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 148 rows = 44 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-03-30 Mon 04:13:56:750.645 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530373604928, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-03-30 Mon 04:13:59:864.787 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.249 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-03-30 Mon 04:13:59:865.029 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID e0c8041b-2bee-11f1-92ac-00000a0000f9 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-03-30 Mon 04:13:59:865.085 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QEQueryLog: e0c8041b-2bee-11f1-92ac-00000a0000f9 :0:json query is: { end_time: 1774844039845341 , enqueue_time: 1774844039864363 , query_metadata: {"enqueue_time": 1774844039864220} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "301c349e-4d87-4414-8c1f-0ca1dc98e33c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:13:59:865.251 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-03-30 Mon 04:13:59:865.268 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-03-30 Mon 04:13:59:865.314 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for e0c8041b-2bee-11f1-92ac-00000a0000f9 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-03-30 Mon 04:13:59:865.500 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: [SYS_INFO]: QEQueryLog: e0c8041b-2bee-11f1-92ac-00000a0000f9 :0:json query is: { end_time: 1774844039845341 , enqueue_time: 1774844039864363 , query_metadata: {"enqueue_time": 1774844039864220} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "301c349e-4d87-4414-8c1f-0ca1dc98e33c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:13:59:916.903 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:916.962 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:916.973 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:917.016 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:917.040 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:917.056 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:917.079 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:917.093 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:917.103 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 5 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:917.114 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 5 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:917.476 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QEQueryLog: e0c8041b-2bee-11f1-92ac-00000a0000f9 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 301c349e-4d87-4414-8c1f-0ca1dc98e33c src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:59:917.590 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QEQueryLog: e0c8041b-2bee-11f1-92ac-00000a0000f9 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 301c349e-4d87-4414-8c1f-0ca1dc98e33c src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:59:917.708 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QEQueryLog: e0c8041b-2bee-11f1-92ac-00000a0000f9 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 301c349e-4d87-4414-8c1f-0ca1dc98e33c src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:59:917.834 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QEQueryLog: e0c8041b-2bee-11f1-92ac-00000a0000f9 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 301c349e-4d87-4414-8c1f-0ca1dc98e33c src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:59:917.861 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530476045888, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:917.898 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530476045888, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:917.906 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530476045888, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:917.912 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530476045888, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:917.927 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530476045888, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:917.933 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530476045888, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:917.939 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530476045888, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:917.944 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530476045888, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:917.950 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530476045888, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:917.956 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530476045888, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:917.970 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QEQueryLog: e0c8041b-2bee-11f1-92ac-00000a0000f9 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 301c349e-4d87-4414-8c1f-0ca1dc98e33c src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-30 Mon 04:13:59:918.058 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:918.066 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:918.073 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:918.123 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:918.133 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:918.139 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:918.146 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:918.155 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:918.171 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-30 Mon 04:13:59:918.180 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530467649088, Pid 1]: [SYS_INFO]: QELog: e0c8041b-2bee-11f1-92ac-00000a0000f9 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-30 Mon 04:13:59:918.278 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QEQueryLog: e0c8041b-2bee-11f1-92ac-00000a0000f9 :0:json query is: { end_time: 1774844039845341 , enqueue_time: 1774844039864363 , query_metadata: {"enqueue_time": 1774844039864220} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774844018247385 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "301c349e-4d87-4414-8c1f-0ca1dc98e33c", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-30 Mon 04:13:59:918.457 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-03-30 Mon 04:13:59:918.504 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-03-30 Mon 04:13:59:918.567 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-03-30 Mon 04:13:59:919.187 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QELog: Finished: QID e0c8041b-2bee-11f1-92ac-00000a0000f9 Table ObjectVMTable Time(ms) 54 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-03-30 Mon 04:13:59:919.285 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-03-30 Mon 04:13:59:919.339 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5563-1. table = ObjectVMTable query_stats= [ qid = e0c8041b-2bee-11f1-92ac-00000a0000f9 where = [[{"name": "ObjectId", "value": "301c349e-4d87-4414-8c1f-0ca1dc98e33c", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 52,0, 51,0, 51,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 54 rows = 5 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-03-30 Mon 04:14:12:413.283 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530365208128, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2026-03-30 Mon 04:15:12:413.826 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530386200128, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2026-03-30 Mon 04:16:12:414.611 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529379567168, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2026-03-30 Mon 04:17:12:415.187 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530373604928, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2026-03-30 Mon 04:18:12:415.882 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530382001728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2026-03-30 Mon 04:19:12:416.649 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2026-03-30 Mon 04:20:12:417.273 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140529375368768, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2026-03-30 Mon 04:21:12:417.909 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530377803328, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2026-03-30 Mon 04:22:12:419.131 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530463450688, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2026-03-30 Mon 04:23:12:420.677 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530361009728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2026-03-30 Mon 04:24:12:421.446 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530369406528, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2026-03-30 Mon 04:25:12:422.128 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530377803328, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2026-03-30 Mon 04:25:39:665.700 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:8091::10.0.0.249:33752 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-30 Mon 04:26:12:422.758 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530382001728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2026-03-30 Mon 04:27:12:423.815 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530377803328, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2026-03-30 Mon 04:27:56:785.839 UTC cn-jenkins-deploy-platform-ansible-os-5563-1 [Thread 140530483185792, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.249:8091::10.0.0.249:43388 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256