2026-03-26 Thu 05:28:14:739.456 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: http-server-port 8091 2026-03-26 Thu 05:28:14:739.561 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Max-tasks 16 2026-03-26 Thu 05:28:14:739.570 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Max-slice 100 2026-03-26 Thu 05:28:14:739.577 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Collectors 10.0.0.38:8086 2026-03-26 Thu 05:28:14:739.583 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Collectors 10.0.0.242:8086 2026-03-26 Thu 05:28:14:739.589 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Collectors 10.0.0.33:8086 2026-03-26 Thu 05:28:14:739.610 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: ROLE : Generator 2026-03-26 Thu 05:28:14:739.617 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-26 Thu 05:28:14:739.623 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5546-3. 2026-03-26 Thu 05:28:14:739.629 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-26 Thu 05:28:14:739.635 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-26 Thu 05:28:14:739.641 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-26 Thu 05:28:14:739.648 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-26 Thu 05:28:14:740.644 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: HTTP Introspect Init 2026-03-26 Thu 05:28:14:740.764 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-03-26 Thu 05:28:14:740.831 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Sandesh Http Server Port: 8091 2026-03-26 Thu 05:28:14:740.874 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-26 Thu 05:28:14:740.884 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-26 Thu 05:28:14:740.964 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Idle 2026-03-26 Thu 05:28:14:741.133 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: No Client: 1774502894741051 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774502894740976 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:28:14:741.293 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-26 Thu 05:28:14:741.309 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-26 Thu 05:28:14:741.323 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.242:9042 10.0.0.33:9042 2026-03-26 Thu 05:28:14:741.488 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Processing scm::EvStart in state Idle 2026-03-26 Thu 05:28:14:741.594 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Disconnect 2026-03-26 Thu 05:28:14:741.710 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-26 Thu 05:28:14:742.067 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-26 Thu 05:28:14:742.193 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:28:14:742.228 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: SANDESH: Send FAILED: 1774502894741580 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:28:14:742.242 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:28:14:742.256 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:28:14:742.390 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: SANDESH: Send FAILED: 1774502894741692 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774502894740976 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:28:14:742.423 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:28:14:742.453 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:28:14:742.580 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: SANDESH: Send FAILED: 1774502894741797 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774502894740976 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:28:14:742.604 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:28:14:742.621 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:28:14:742.646 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: SANDESH: Send FAILED: 1774502894741849 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:28:14:742.657 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:28:14:742.670 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:28:14:742.779 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: SANDESH: Send FAILED: 1774502894742173 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774502894740976 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:28:14:742.797 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722594592512, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:28:14:754.095 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-26 Thu 05:28:14:754.144 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-26 Thu 05:28:14:754.224 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-26 Thu 05:28:14:754.302 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722551944960, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:28:14:754.338 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722551944960, Pid 1]: SANDESH: Send FAILED: 1774502894754188 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:28:14:754.349 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722551944960, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:28:19:761.933 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-26 Thu 05:28:19:761.979 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-26 Thu 05:28:19:762.021 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-26 Thu 05:28:24:764.058 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-26 Thu 05:28:24:764.109 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-26 Thu 05:28:24:764.135 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-26 Thu 05:28:29:765.975 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-26 Thu 05:28:29:766.030 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-26 Thu 05:28:29:766.058 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-26 Thu 05:28:34:994.407 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-26 Thu 05:28:34:994.480 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-26 Thu 05:28:34:994.508 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-26 Thu 05:28:40:066.724 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-26 Thu 05:28:40:066.779 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-26 Thu 05:28:40:066.804 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-26 Thu 05:28:45:089.799 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-26 Thu 05:28:45:089.876 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-26 Thu 05:28:45:089.902 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-26 Thu 05:28:50:138.881 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-26 Thu 05:28:50:138.982 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-26 Thu 05:28:50:139.012 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-26 Thu 05:28:55:197.718 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-26 Thu 05:28:55:197.803 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-26 Thu 05:29:00:227.131 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-26 Thu 05:29:05:234.407 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-26 Thu 05:29:05:234.468 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: -(-1):EvTcpConnectFail 2026-03-26 Thu 05:29:05:234.490 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x11fd5f0 2026-03-26 Thu 05:29:05:234.497 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.513 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.528 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x11fdca0 2026-03-26 Thu 05:29:05:234.535 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.545 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.556 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x11f71d0 2026-03-26 Thu 05:29:05:234.562 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.571 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.581 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x11fe340 2026-03-26 Thu 05:29:05:234.588 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.597 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.607 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x11fe870 2026-03-26 Thu 05:29:05:234.613 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.622 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.643 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x1201fe0 2026-03-26 Thu 05:29:05:234.649 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.659 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.669 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x1202920 2026-03-26 Thu 05:29:05:234.675 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.687 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.697 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x1203260 2026-03-26 Thu 05:29:05:234.703 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.713 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.723 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x1203ba0 2026-03-26 Thu 05:29:05:234.729 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.738 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.748 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x12044e0 2026-03-26 Thu 05:29:05:234.754 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.763 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.773 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x11ff1a0 2026-03-26 Thu 05:29:05:234.779 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.788 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.797 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x11ffae0 2026-03-26 Thu 05:29:05:234.803 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.812 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.822 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x1200420 2026-03-26 Thu 05:29:05:234.828 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.840 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.849 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x1200d60 2026-03-26 Thu 05:29:05:234.855 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.864 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:234.874 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: 0 0x12016a0 2026-03-26 Thu 05:29:05:234.880 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:29:05:234.889 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:29:05:235.175 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:29:05:235.222 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: SANDESH: Send FAILED: 1774502945234252 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:29:05:235.233 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:29:05:235.246 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-26 Thu 05:29:05:235.255 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Connect : EvTcpConnectFail 2026-03-26 Thu 05:29:05:235.340 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:235.389 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:235.410 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:235.462 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:235.514 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:235.526 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:235.553 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:235.564 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:235.578 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:235.589 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:235.657 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:235.707 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:235.730 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:235.749 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:235.767 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:235.812 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:235.825 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:235.841 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:235.851 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:235.865 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:235.876 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:235.890 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:235.901 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:235.921 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:235.932 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:235.954 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:235.990 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:236.009 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:236.026 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:236.043 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:235.414 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-26 Thu 05:29:05:236.288 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Idle 2026-03-26 Thu 05:29:05:236.381 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:29:05:236.518 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: SANDESH: Send FAILED: 1774502945235361 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774502894740976 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 43 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 44, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:29:05:236.540 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:29:05:236.553 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:29:05:236.575 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: SANDESH: Send FAILED: 1774502945235393 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:29:05:236.584 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:29:05:236.594 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:29:05:236.615 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: SANDESH: Send FAILED: 1774502945235639 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:29:05:236.626 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:29:05:236.636 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:29:05:236.656 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: SANDESH: Send FAILED: 1774502945235699 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.242:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:29:05:236.663 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:29:05:236.673 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:29:05:236.693 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: SANDESH: Send FAILED: 1774502945235977 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:29:05:236.700 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:29:05:236.710 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-26 Thu 05:29:05:236.757 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:29:05:236.922 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: SANDESH: Send FAILED: 1774502945236366 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774502894740976 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 72 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 74, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:29:05:236.948 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:29:05:237.067 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:29:05:237.128 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SANDESH: Send FAILED: 1774502945236972 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774502945236286 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774502945236754 ] max_sm_queue_count = 7 ] 2026-03-26 Thu 05:29:05:237.139 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:29:05:237.149 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:29:05:237.277 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SANDESH: Send FAILED: 1774502945237045 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774502894740976 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 72 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 74, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 13, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:29:05:237.297 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:29:05:237.384 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:237.400 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:237.417 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:237.428 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:237.443 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:237.466 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:237.481 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:237.491 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:237.505 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:29:05:237.516 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:29:05:237.538 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:237.586 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:237.605 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:237.610 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722581997312, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:29:05:237.623 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:237.643 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:29:05:237.653 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722581997312, Pid 1]: SANDESH: Send FAILED: 1774502945237563 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:29:05:237.663 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722581997312, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:29:06:236.410 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-26 Thu 05:29:06:236.672 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Connect : Start Connect timer 10.0.0.242:8086 2026-03-26 Thu 05:29:06:236.797 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:29:06:236.842 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SANDESH: Send FAILED: 1774502946236502 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.242:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:29:06:236.852 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:29:06:236.863 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:29:06:236.994 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SANDESH: Send FAILED: 1774502946236780 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774502894740976 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:29:06:237.010 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:29:06:240.779 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:46702::10.0.0.242:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-26 Thu 05:29:06:240.825 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: 10.0.0.33:46702::10.0.0.242:8086(-1) OnSessionEvent TCP Connected 2026-03-26 Thu 05:29:06:240.839 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: 10.0.0.33:46702::10.0.0.242:8086(-1):EvTcpConnected 2026-03-26 Thu 05:29:06:240.908 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-26 Thu 05:29:06:240.940 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-26 Thu 05:29:06:240.985 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: ClientInit 2026-03-26 Thu 05:29:06:241.012 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5546-3.:contrail-query-engine:0:Database count 1 2026-03-26 Thu 05:29:06:241.107 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:29:06:241.119 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:29:06:241.147 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: SANDESH: Send FAILED: 1774502946240977 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.242:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:29:06:241.164 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:29:06:241.175 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:29:06:241.184 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-26 Thu 05:29:06:241.198 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:29:06:241.207 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-26 Thu 05:29:06:241.328 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: SANDESH: Send FAILED: 1774502946241094 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774502894740976 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:29:06:241.345 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722590394112, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-26 Thu 05:29:06:243.926 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: OnMessage control in state: ClientInit 2026-03-26 Thu 05:29:06:243.989 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-26 Thu 05:29:06:244.005 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-26 Thu 05:29:06:244.056 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Received Ctrl Message with size 0 2026-03-26 Thu 05:29:06:245.065 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774502894740976 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 17 dequeues = 17 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774502945236286 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774502945236754 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 87 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 7 proxy seq 10 2026-03-26 Thu 05:29:06:245.123 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-26 Thu 05:29:06:245.164 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.242:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2026-03-26 Thu 05:29:06:245.182 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-26 Thu 05:29:06:245.193 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-26 Thu 05:29:06:245.205 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-26 Thu 05:29:06:245.254 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Established 2026-03-26 Thu 05:29:13:124.725 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722551944960, Pid 1]: OnMessage control in state: Established 2026-03-26 Thu 05:29:13:124.803 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722551944960, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-03-26 Thu 05:29:13:124.815 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722551944960, Pid 1]: Established : EvSandeshMessageRecv 2026-03-26 Thu 05:29:13:124.864 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722551944960, Pid 1]: Received Ctrl Message with size 3 2026-03-26 Thu 05:29:13:124.880 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722551944960, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2026-03-26 Thu 05:29:13:124.896 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722551944960, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2026-03-26 Thu 05:29:13:124.911 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722551944960, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2026-03-26 Thu 05:29:13:124.920 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722551944960, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-26 Thu 05:29:35:237.938 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722131240704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-26 Thu 05:30:35:239.536 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722577798912, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-26 Thu 05:31:35:241.341 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722573600512, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-26 Thu 05:32:35:242.055 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722543548160, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-26 Thu 05:33:35:243.609 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-26 Thu 05:33:35:972.543 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.33:37806 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-26 Thu 05:34:03:184.500 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.33:56702 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-26 Thu 05:34:35:245.342 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-26 Thu 05:35:35:247.923 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722131240704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-26 Thu 05:36:22:228.394 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Disconnect status: -1 0x12044e0 2026-03-26 Thu 05:36:22:228.458 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:36:22:228.624 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Disconnect status: -1 0x1203ba0 2026-03-26 Thu 05:36:22:228.643 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:36:22:228.675 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Disconnect status: -1 0x1203260 2026-03-26 Thu 05:36:22:228.687 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:36:22:228.715 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Disconnect status: -1 0x1202920 2026-03-26 Thu 05:36:22:228.726 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:36:22:228.754 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-26 Thu 05:36:22:228.763 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Disconnect status: -1 0x1201fe0 2026-03-26 Thu 05:36:22:228.773 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:36:22:234.627 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: -1 0x12044e0 2026-03-26 Thu 05:36:22:234.712 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: -1 0x1203ba0 2026-03-26 Thu 05:36:22:234.731 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: -1 0x1203260 2026-03-26 Thu 05:36:22:234.745 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: -1 0x1202920 2026-03-26 Thu 05:36:22:234.767 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722742462592, Pid 1]: RAC_Connect status: -1 0x1201fe0 2026-03-26 Thu 05:36:22:325.755 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Processing scm::EvStop in state Established 2026-03-26 Thu 05:36:22:326.314 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Idle 2026-03-26 Thu 05:36:22:326.461 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-26 Thu 05:36:22:326.477 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:36:22:326.662 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: SANDESH: Send FAILED: 1774503382326445 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774502894740976 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, (*_iter40) = 10.0.0.38:8086, ] ] rx_socket_stats= [ bytes = 2446 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 382208 calls = 42 average_bytes = 9100 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 93 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 30 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 43, ] ] ] 2026-03-26 Thu 05:36:22:326.687 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139722586195712, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:36:36:900.408 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: http-server-port 8091 2026-03-26 Thu 05:36:36:900.506 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Max-tasks 16 2026-03-26 Thu 05:36:36:900.516 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Max-slice 100 2026-03-26 Thu 05:36:36:900.523 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Collectors 10.0.0.38:8086 2026-03-26 Thu 05:36:36:900.534 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Collectors 10.0.0.242:8086 2026-03-26 Thu 05:36:36:900.540 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Collectors 10.0.0.33:8086 2026-03-26 Thu 05:36:36:900.573 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: ROLE : Generator 2026-03-26 Thu 05:36:36:900.580 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-26 Thu 05:36:36:900.587 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5546-3. 2026-03-26 Thu 05:36:36:900.593 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-26 Thu 05:36:36:900.599 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-26 Thu 05:36:36:900.605 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-26 Thu 05:36:36:900.611 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-26 Thu 05:36:36:901.256 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: HTTP Introspect Init 2026-03-26 Thu 05:36:36:927.273 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-03-26 Thu 05:36:36:927.760 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Sandesh Http Server Port: 8091 2026-03-26 Thu 05:36:36:927.831 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-26 Thu 05:36:36:927.848 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-26 Thu 05:36:36:931.238 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Idle 2026-03-26 Thu 05:36:36:931.541 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: No Client: 1774503396931409 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:36:36:931.746 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-26 Thu 05:36:36:931.772 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-26 Thu 05:36:36:931.796 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.242:9042 10.0.0.33:9042 2026-03-26 Thu 05:36:36:932.362 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Processing scm::EvStart in state Idle 2026-03-26 Thu 05:36:36:932.435 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Disconnect 2026-03-26 Thu 05:36:36:932.512 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-26 Thu 05:36:36:932.765 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-26 Thu 05:36:36:932.845 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:36:36:932.868 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: SANDESH: Send FAILED: 1774503396932427 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:36:36:932.879 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:36:36:932.890 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:36:36:932.979 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: SANDESH: Send FAILED: 1774503396932501 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:36:36:932.997 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:36:36:933.008 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:36:36:933.092 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: SANDESH: Send FAILED: 1774503396932579 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:36:36:933.107 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:36:36:933.118 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:36:36:933.135 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: SANDESH: Send FAILED: 1774503396932601 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:36:36:933.143 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:36:36:933.151 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:36:36:933.222 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: SANDESH: Send FAILED: 1774503396932832 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:36:36:933.236 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:36:37:103.303 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-26 Thu 05:36:37:114.336 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:36:37:114.423 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: SANDESH: Send FAILED: 1774503397113389 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:36:37:114.440 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:36:37:116.793 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:58812::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-26 Thu 05:36:37:116.833 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:58812::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-03-26 Thu 05:36:37:116.843 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:58812::10.0.0.38:8086(-1):EvTcpConnected 2026-03-26 Thu 05:36:37:116.879 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x12205f0 2026-03-26 Thu 05:36:37:116.887 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:37:116.901 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:37:116.917 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1220ca0 2026-03-26 Thu 05:36:37:116.924 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:37:116.940 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:37:116.952 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x121a1d0 2026-03-26 Thu 05:36:37:116.958 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:37:116.968 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:37:116.978 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1221340 2026-03-26 Thu 05:36:37:116.984 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:37:116.993 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:37:117.004 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1221870 2026-03-26 Thu 05:36:37:117.010 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:37:117.019 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:37:117.040 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:36:37:117.081 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:36:37:117.100 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:36:37:117.117 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:36:37:117.133 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:36:37:117.148 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1224fe0 2026-03-26 Thu 05:36:37:117.154 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:37:117.164 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:37:117.174 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1225920 2026-03-26 Thu 05:36:37:117.180 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:37:117.188 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:37:117.198 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1226260 2026-03-26 Thu 05:36:37:117.204 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:37:117.212 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:37:117.226 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1226ba0 2026-03-26 Thu 05:36:37:117.232 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:37:117.241 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:37:117.251 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x12274e0 2026-03-26 Thu 05:36:37:117.257 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:37:117.266 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:37:117.558 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-26 Thu 05:36:37:117.597 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-26 Thu 05:36:37:117.688 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: ClientInit 2026-03-26 Thu 05:36:37:117.709 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5546-3.:contrail-query-engine:0:Database count 1 2026-03-26 Thu 05:36:37:117.896 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:36:37:117.931 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:36:37:117.970 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: SANDESH: Send FAILED: 1774503397117681 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:36:37:117.984 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:36:37:118.001 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:36:37:118.014 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-26 Thu 05:36:37:118.033 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:36:37:118.045 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-26 Thu 05:36:37:118.082 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:37:118.106 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:37:118.144 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:37:118.157 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:37:118.176 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:37:118.189 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:37:118.206 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:37:118.219 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:37:118.237 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:37:118.251 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:37:118.268 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:37:118.280 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:37:118.295 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:37:118.298 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: SANDESH: Send FAILED: 1774503397117799 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 13, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:36:37:118.361 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-26 Thu 05:36:37:135.638 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:37:135.702 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:37:135.725 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:37:135.741 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:37:135.752 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:37:135.767 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:37:135.777 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:37:135.866 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:37:135.901 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:37:135.922 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:37:135.941 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:37:135.960 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:37:135.989 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:37:136.073 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:37:136.093 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:37:136.109 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:37:136.126 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:37:136.252 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:36:37:136.268 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:36:37:136.296 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: SANDESH: Send FAILED: 1774503397135827 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:36:37:136.314 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:36:37:136.325 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:36:37:136.333 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:36:37:136.354 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: SANDESH: Send FAILED: 1774503397135894 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.242:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:36:37:136.361 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:36:37:136.370 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:36:37:136.378 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:36:37:136.397 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: SANDESH: Send FAILED: 1774503397136065 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.242:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:36:37:136.404 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:36:37:142.035 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: OnMessage control in state: ClientInit 2026-03-26 Thu 05:36:37:142.091 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-26 Thu 05:36:37:142.106 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-26 Thu 05:36:37:142.149 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Received Ctrl Message with size 0 2026-03-26 Thu 05:36:37:142.289 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 13, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2026-03-26 Thu 05:36:37:142.315 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-26 Thu 05:36:37:142.344 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.242:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 7 2026-03-26 Thu 05:36:37:142.359 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-26 Thu 05:36:37:142.367 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-26 Thu 05:36:37:142.376 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-26 Thu 05:36:37:142.412 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535471724288, Pid 1]: Established 2026-03-26 Thu 05:36:42:117.099 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:36:42:117.216 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:36:42:117.257 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:36:42:117.291 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:36:42:117.323 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:36:42:117.679 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x12221a0 2026-03-26 Thu 05:36:42:117.696 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:42:117.724 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:42:117.747 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1222ae0 2026-03-26 Thu 05:36:42:117.753 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:42:117.764 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:42:117.774 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1223420 2026-03-26 Thu 05:36:42:117.780 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:42:117.788 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:42:117.798 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1223d60 2026-03-26 Thu 05:36:42:117.804 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:42:117.812 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:42:117.839 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x12246a0 2026-03-26 Thu 05:36:42:117.847 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:36:42:117.856 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:36:42:118.272 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:42:118.296 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:42:118.313 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:42:118.324 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:42:118.338 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:42:118.349 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:42:118.363 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:42:118.374 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:42:118.409 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:42:118.465 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:42:118.483 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:42:118.499 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:36:42:118.516 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:36:42:118.527 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:36:42:118.544 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:37:36:935.231 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-26 Thu 05:38:36:935.833 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-26 Thu 05:39:36:936.686 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-26 Thu 05:39:43:501.106 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x12274e0 2026-03-26 Thu 05:39:43:501.174 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:501.219 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x1226ba0 2026-03-26 Thu 05:39:43:501.233 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:501.260 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x1226260 2026-03-26 Thu 05:39:43:501.272 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:501.306 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x1225920 2026-03-26 Thu 05:39:43:501.317 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:501.346 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-26 Thu 05:39:43:501.362 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x1224fe0 2026-03-26 Thu 05:39:43:501.379 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:501.681 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:39:43:501.713 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:39:43:501.729 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:39:43:501.743 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:39:43:501.757 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:39:43:618.416 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x1221870 2026-03-26 Thu 05:39:43:618.489 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:618.689 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x1221340 2026-03-26 Thu 05:39:43:618.712 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:618.750 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x121a1d0 2026-03-26 Thu 05:39:43:618.766 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:618.808 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x1220ca0 2026-03-26 Thu 05:39:43:618.823 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:618.850 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-26 Thu 05:39:43:618.861 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x12205f0 2026-03-26 Thu 05:39:43:618.871 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:619.228 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:39:43:619.286 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:39:43:619.308 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:39:43:619.323 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:39:43:619.338 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:39:43:624.924 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x1223d60 2026-03-26 Thu 05:39:43:624.976 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:625.114 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x12246a0 2026-03-26 Thu 05:39:43:625.131 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:625.164 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x1222ae0 2026-03-26 Thu 05:39:43:625.178 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:625.210 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-26 Thu 05:39:43:625.218 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x12221a0 2026-03-26 Thu 05:39:43:625.228 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:625.249 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Disconnect status: -1 0x1223420 2026-03-26 Thu 05:39:43:625.262 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-26 Thu 05:39:43:625.567 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:39:43:625.603 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:39:43:625.661 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:39:43:625.680 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:39:43:625.717 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:39:43:676.474 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.33:58812::10.0.0.38:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-26 Thu 05:39:43:688.324 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:58812::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-26 Thu 05:39:43:688.348 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:58812::10.0.0.38:8086(-1):EvTcpClose 2026-03-26 Thu 05:39:43:688.402 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvTcpClose in state Established 2026-03-26 Thu 05:39:43:688.415 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Established : EvTcpClose 2026-03-26 Thu 05:39:43:691.024 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:34996::10.0.0.33:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-26 Thu 05:39:43:691.061 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:34996::10.0.0.33:8086(-1) OnSessionEvent TCP Connected 2026-03-26 Thu 05:39:43:691.070 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:34996::10.0.0.33:8086(-1):EvTcpConnected 2026-03-26 Thu 05:39:43:691.111 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-26 Thu 05:39:43:691.248 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:39:43:691.430 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503583688512 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = cn-jenkins-deploy-platform-ansible-os-5546-1. collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 190188 calls = 25 average_bytes = 7607 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 82, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 11, _iter59->first = sent _iter59->second = 26, ] ] ] 2026-03-26 Thu 05:39:43:691.450 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:39:43:691.463 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:39:43:691.489 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503583688546 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:39:43:691.500 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:39:43:691.511 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2026-03-26 Thu 05:39:43:691.566 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-26 Thu 05:39:43:691.575 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-26 Thu 05:39:43:691.605 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: ClientInit 2026-03-26 Thu 05:39:43:692.208 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5546-3.:contrail-query-engine:0:Database count 2 2026-03-26 Thu 05:39:43:692.292 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:39:43:692.302 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-26 Thu 05:39:43:692.464 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503583691225 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 190188 calls = 25 average_bytes = 7607 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 11, _iter59->first = sent _iter59->second = 26, ] ] ] 2026-03-26 Thu 05:39:43:692.485 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-26 Thu 05:39:43:692.496 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:39:43:692.505 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:39:43:692.528 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503583691599 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:39:43:692.535 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:39:43:692.544 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:39:43:692.563 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-26 Thu 05:39:43:692.581 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:39:43:692.593 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-26 Thu 05:39:43:692.786 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503583692282 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 190188 calls = 25 average_bytes = 7607 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 13, _iter59->first = sent _iter59->second = 26, ] ] ] 2026-03-26 Thu 05:39:43:692.805 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-26 Thu 05:39:43:791.874 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:34996::10.0.0.33:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-26 Thu 05:39:43:791.905 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:34996::10.0.0.33:8086(-1):EvTcpClose 2026-03-26 Thu 05:39:43:791.962 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-26 Thu 05:39:43:791.980 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: ClientInit : EvTcpClose 2026-03-26 Thu 05:39:43:792.125 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Return to idle with 5000 2026-03-26 Thu 05:39:43:792.136 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Idle 2026-03-26 Thu 05:39:43:792.202 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:43:792.371 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: SANDESH: Send FAILED: 1774503583792077 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 191761 calls = 26 average_bytes = 7375 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-03-26 Thu 05:39:43:792.395 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:39:43:792.407 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:43:792.438 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: SANDESH: Send FAILED: 1774503583792110 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:39:43:792.446 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:39:43:792.456 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-26 Thu 05:39:43:792.487 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:43:792.650 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: SANDESH: Send FAILED: 1774503583792192 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 191761 calls = 26 average_bytes = 7375 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-03-26 Thu 05:39:43:792.673 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:39:48:501.763 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:39:48:501.954 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:39:48:502.019 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:39:48:502.049 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:39:48:502.062 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:39:48:502.094 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:39:48:502.124 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:39:48:502.179 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:39:48:502.198 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:39:48:502.212 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:39:48:619.304 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:39:48:619.437 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:39:48:619.486 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:39:48:619.523 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:39:48:619.567 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:39:48:619.641 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:39:48:619.671 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:39:48:619.694 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:39:48:619.724 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:39:48:619.786 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:39:48:625.660 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:39:48:625.785 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:39:48:625.845 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:39:48:625.880 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:39:48:625.917 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:39:48:626.079 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:39:48:626.112 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:39:48:626.412 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:39:48:626.451 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:39:48:626.467 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:39:48:792.262 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-26 Thu 05:39:48:792.580 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Connect : Start Connect timer 10.0.0.242:8086 2026-03-26 Thu 05:39:48:792.764 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:39:48:792.812 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: SANDESH: Send FAILED: 1774503588792362 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.242:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:39:48:792.826 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:39:48:792.843 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:39:48:793.153 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: SANDESH: Send FAILED: 1774503588792738 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 191761 calls = 26 average_bytes = 7375 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 19, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-03-26 Thu 05:39:48:793.191 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:39:48:798.095 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:46576::10.0.0.242:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-26 Thu 05:39:48:798.134 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:46576::10.0.0.242:8086(-1) OnSessionEvent TCP Connected 2026-03-26 Thu 05:39:48:798.145 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:46576::10.0.0.242:8086(-1):EvTcpConnected 2026-03-26 Thu 05:39:48:798.196 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-26 Thu 05:39:48:798.209 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-26 Thu 05:39:48:798.248 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: ClientInit 2026-03-26 Thu 05:39:48:798.270 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5546-3.:contrail-query-engine:0:Database count 3 2026-03-26 Thu 05:39:48:798.389 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:39:48:798.402 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:39:48:798.429 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: SANDESH: Send FAILED: 1774503588798241 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.242:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:39:48:798.447 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:39:48:798.459 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:39:48:798.467 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-26 Thu 05:39:48:798.480 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:39:48:798.487 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-26 Thu 05:39:48:798.667 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: SANDESH: Send FAILED: 1774503588798377 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 191761 calls = 26 average_bytes = 7375 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 21, _iter59->first = sent _iter59->second = 27, ] ] ] 2026-03-26 Thu 05:39:48:798.688 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-26 Thu 05:39:50:798.396 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: 10.0.0.242:8086 EvConnectTimerExpired in state ClientInit 2026-03-26 Thu 05:39:50:798.474 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2026-03-26 Thu 05:39:50:798.486 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: ClientInit : EvConnectTimerExpired 2026-03-26 Thu 05:39:50:798.726 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Return to idle with 5000 2026-03-26 Thu 05:39:50:798.740 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Idle 2026-03-26 Thu 05:39:50:798.811 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:50:798.993 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: SANDESH: Send FAILED: 1774503590798597 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:39:50:799.021 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:39:50:799.033 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:50:799.059 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: SANDESH: Send FAILED: 1774503590798637 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.242:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:39:50:799.071 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:39:50:799.085 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-26 Thu 05:39:50:799.121 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:50:799.289 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: SANDESH: Send FAILED: 1774503590798800 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:39:50:799.311 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:39:53:502.095 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:39:53:502.292 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:39:53:502.356 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:39:53:502.388 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:39:53:502.401 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:39:53:502.434 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:39:53:502.467 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:39:53:502.503 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:39:53:502.520 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:39:53:502.534 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:39:53:619.727 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:39:53:619.910 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:39:53:619.966 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:39:53:620.002 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:39:53:620.034 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:39:53:620.077 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:39:53:620.110 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:39:53:620.127 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:39:53:620.141 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:39:53:620.155 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:39:53:626.141 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:39:53:626.264 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:39:53:626.453 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:39:53:626.511 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:39:53:626.544 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:39:53:626.710 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:39:53:626.794 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:39:53:626.824 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:39:53:626.850 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:39:53:626.887 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:39:55:798.879 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-26 Thu 05:39:55:799.140 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-26 Thu 05:39:55:799.280 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:39:55:799.314 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SANDESH: Send FAILED: 1774503595798961 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:39:55:799.325 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:39:55:799.335 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:39:55:799.496 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SANDESH: Send FAILED: 1774503595799261 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 26, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:39:55:799.521 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:39:55:799.562 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-26 Thu 05:39:55:799.591 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1):EvTcpConnectFail 2026-03-26 Thu 05:39:55:799.628 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-26 Thu 05:39:55:799.640 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Connect : EvTcpConnectFail 2026-03-26 Thu 05:39:55:799.769 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-26 Thu 05:39:55:799.804 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Idle 2026-03-26 Thu 05:39:55:799.875 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:55:800.041 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: SANDESH: Send FAILED: 1774503595799722 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:39:55:800.068 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:39:55:800.080 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:55:800.105 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: SANDESH: Send FAILED: 1774503595799750 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:39:55:800.114 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:39:55:800.123 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-26 Thu 05:39:55:800.147 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:55:800.296 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: SANDESH: Send FAILED: 1774503595799864 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:39:55:800.315 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535450732288, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:39:58:502.452 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:39:58:502.613 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:39:58:502.688 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:39:58:502.722 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:39:58:502.735 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:39:58:502.770 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:39:58:502.801 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:39:58:502.838 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:39:58:502.855 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:39:58:502.870 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:39:58:620.181 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:39:58:637.565 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:39:58:637.672 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:39:58:637.772 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:39:58:637.818 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:39:58:637.903 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:39:58:637.966 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:39:58:638.015 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:39:58:638.063 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:39:58:638.127 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:39:58:638.178 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:39:58:638.224 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:39:58:638.289 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:39:58:638.320 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:39:58:638.343 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:39:58:638.366 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:39:58:638.433 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:39:58:638.500 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:39:58:638.523 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:39:58:638.539 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:39:59:800.019 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-26 Thu 05:39:59:800.347 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-26 Thu 05:39:59:800.523 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:39:59:800.589 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: SANDESH: Send FAILED: 1774503599800113 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:39:59:800.607 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:39:59:800.610 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-26 Thu 05:39:59:800.662 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1):EvTcpConnectFail 2026-03-26 Thu 05:39:59:800.631 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:39:59:800.858 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: SANDESH: Send FAILED: 1774503599800501 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 31, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:39:59:800.885 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:39:59:800.897 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-26 Thu 05:39:59:800.906 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Connect : EvTcpConnectFail 2026-03-26 Thu 05:39:59:801.026 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-26 Thu 05:39:59:801.060 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Idle 2026-03-26 Thu 05:39:59:801.133 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:59:801.300 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: SANDESH: Send FAILED: 1774503599800974 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:39:59:801.318 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:39:59:801.330 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:59:801.354 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: SANDESH: Send FAILED: 1774503599800998 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:39:59:801.366 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:39:59:801.379 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-26 Thu 05:39:59:801.410 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:39:59:801.601 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: SANDESH: Send FAILED: 1774503599801116 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:39:59:801.622 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:03:502.761 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:03:502.927 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:03:502.988 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:03:503.019 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:03:503.033 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:03:503.066 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:03:503.097 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:03:503.138 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:03:503.155 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:03:503.169 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:03:637.853 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:03:638.009 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:03:638.171 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:03:638.205 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:03:638.324 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:03:638.389 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:03:638.430 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:03:638.490 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:03:638.543 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:03:638.579 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:03:638.593 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:03:638.605 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:03:638.636 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:03:638.666 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:03:638.699 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:03:638.857 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:03:638.940 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:03:638.973 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:40:03:638.989 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:03:639.004 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:06:938.057 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:06:938.178 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: SANDESH: Send FAILED: 1774503606937909 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 64 dequeues = 64 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 83 dequeues = 83 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774503599801058 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774503599801407 ] max_sm_queue_count = 5 ] 2026-03-26 Thu 05:40:06:938.213 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:06:938.224 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:06:938.391 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: SANDESH: Send FAILED: 1774503606938016 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 36, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:40:06:938.414 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:07:801.445 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-26 Thu 05:40:07:801.694 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Connect : Start Connect timer 10.0.0.242:8086 2026-03-26 Thu 05:40:07:801.816 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:40:07:801.863 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SANDESH: Send FAILED: 1774503607801530 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.242:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:40:07:801.881 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:40:07:801.892 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:40:07:802.114 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SANDESH: Send FAILED: 1774503607801803 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:40:07:802.143 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:07:802.206 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-26 Thu 05:40:07:802.222 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1):EvTcpConnectFail 2026-03-26 Thu 05:40:07:802.753 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-26 Thu 05:40:07:802.799 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Connect : EvTcpConnectFail 2026-03-26 Thu 05:40:07:802.933 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-26 Thu 05:40:07:802.974 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Idle 2026-03-26 Thu 05:40:07:803.044 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:07:803.206 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503607802878 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:40:07:803.229 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:07:803.241 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:07:803.267 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503607802908 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:40:07:803.278 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:40:07:803.289 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-26 Thu 05:40:07:803.314 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:07:803.462 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503607803033 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:40:07:803.477 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:08:503.085 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:08:503.310 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:08:503.372 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:08:503.405 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:08:503.417 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:08:503.451 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:08:503.482 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:08:503.517 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:08:503.534 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:08:503.547 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:08:638.253 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:08:638.418 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:08:638.595 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:08:638.641 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:08:638.693 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:08:638.727 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:08:638.739 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:08:638.775 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:08:638.791 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:08:638.886 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:08:638.970 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:08:639.075 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:08:639.138 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:08:639.182 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:08:639.220 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:08:639.368 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:08:639.408 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:08:639.699 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:08:639.731 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:08:639.753 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:40:13:503.471 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:13:503.682 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:13:503.744 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:13:503.778 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:13:503.791 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:13:503.824 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:13:503.854 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:13:503.890 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:13:503.906 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:13:503.920 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:13:638.799 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:13:638.965 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:13:639.013 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:13:639.048 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:13:639.079 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:13:639.189 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:13:639.221 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:13:639.236 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:13:639.250 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:13:639.265 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:13:639.443 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:13:639.715 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:13:639.807 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:13:639.870 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:13:639.920 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:13:639.962 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:13:640.118 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:13:640.146 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:13:640.190 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:13:640.246 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:40:18:503.844 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:18:504.038 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:18:504.100 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:18:504.131 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:18:504.143 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:18:504.176 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:18:504.206 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:18:504.242 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:18:504.259 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:18:504.273 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:18:639.267 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:18:639.431 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:18:639.476 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:18:639.510 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:18:639.582 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:18:639.734 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:18:639.776 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:18:639.804 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:18:639.835 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:18:639.851 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:18:639.994 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:18:640.159 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:18:640.203 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:18:640.237 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:18:640.275 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:18:640.385 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:18:640.488 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:18:640.522 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:18:640.596 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:18:640.613 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:40:23:504.204 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:23:504.407 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:23:504.470 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:23:504.503 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:23:504.517 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:23:504.568 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:23:504.602 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:23:504.639 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:23:504.657 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:23:504.670 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:23:639.821 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:23:640.040 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:23:640.096 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:23:640.140 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:23:640.253 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:23:640.335 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:23:640.389 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:23:640.422 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:23:640.445 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:23:640.475 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:23:640.493 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:23:640.566 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:23:640.619 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:23:640.668 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:23:640.711 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:23:640.931 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:23:640.959 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:23:640.999 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:23:641.023 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:40:23:641.044 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:23:803.186 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-26 Thu 05:40:23:803.444 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-26 Thu 05:40:23:803.560 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:40:23:803.597 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503623803276 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:40:23:803.608 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:40:23:803.631 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:40:23:803.791 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503623803534 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:40:23:803.807 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:23:803.846 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-26 Thu 05:40:23:803.856 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1):EvTcpConnectFail 2026-03-26 Thu 05:40:23:804.341 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-26 Thu 05:40:23:804.352 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Connect : EvTcpConnectFail 2026-03-26 Thu 05:40:23:804.464 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-26 Thu 05:40:23:804.502 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Idle 2026-03-26 Thu 05:40:23:804.583 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:23:804.753 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SANDESH: Send FAILED: 1774503623804416 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:40:23:804.771 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:23:804.783 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:23:804.808 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SANDESH: Send FAILED: 1774503623804440 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:40:23:804.816 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:40:23:804.825 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-26 Thu 05:40:23:804.839 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:23:804.986 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SANDESH: Send FAILED: 1774503623804572 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:40:23:805.006 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:28:504.560 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:28:504.718 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:28:504.779 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:28:504.808 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:28:504.821 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:28:504.856 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:28:504.886 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:28:504.936 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:28:504.953 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:28:504.967 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:28:640.405 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:28:640.594 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:28:640.655 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:28:640.700 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:28:640.737 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:28:640.861 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:28:640.898 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:28:640.915 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:28:640.929 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:28:640.944 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:28:640.974 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:28:641.017 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:28:641.051 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:28:641.086 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:28:641.113 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:28:641.449 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:28:641.479 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:28:641.494 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:28:641.509 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:28:641.523 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:40:33:504.862 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:33:505.028 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:33:505.091 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:33:505.144 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:33:505.163 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:33:505.210 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:33:505.256 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:33:505.311 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:33:505.335 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:33:505.354 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:33:640.954 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:33:641.114 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:33:641.184 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:33:641.264 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:33:641.294 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:33:641.363 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:33:641.399 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:33:641.414 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:33:641.429 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:33:641.446 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:33:641.513 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:33:641.579 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:33:641.610 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:33:641.644 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:33:641.673 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:33:641.909 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:33:641.945 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:33:641.961 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:33:641.975 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:33:642.001 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:40:36:938.471 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-26 Thu 05:40:36:938.886 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:36:938.970 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: SANDESH: Send FAILED: 1774503636938344 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 76 dequeues = 76 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 101 dequeues = 101 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774503623804500 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774503623804837 ] max_sm_queue_count = 5 ] 2026-03-26 Thu 05:40:36:938.984 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:36:938.995 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:36:939.165 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: SANDESH: Send FAILED: 1774503636938453 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 48, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:40:36:939.194 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:38:505.188 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:38:505.385 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:38:505.418 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:38:505.457 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:38:505.489 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:38:505.520 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:38:505.568 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:38:505.586 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:38:505.601 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:38:505.615 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:38:641.442 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:38:641.603 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:38:641.695 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:38:641.730 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:38:641.789 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:38:641.879 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:38:641.920 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:38:641.943 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:38:641.957 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:38:641.998 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:38:642.011 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:38:642.045 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:38:642.071 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:38:642.105 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:38:642.132 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:38:642.351 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:38:642.395 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:38:642.427 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:38:642.446 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:40:38:642.466 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:42:118.146 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:42:118.256 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: SANDESH: Send FAILED: 1774503642118090 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-26 Thu 05:40:42:118.269 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-03-26 Thu 05:40:43:505.482 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:43:505.680 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:43:505.709 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:43:505.747 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:43:505.780 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:43:505.811 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:43:505.876 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:43:505.897 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:43:505.926 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:43:505.940 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:43:641.961 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:43:642.130 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:43:642.171 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:43:642.202 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:43:642.232 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:43:642.314 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:43:642.343 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:43:642.377 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:43:642.391 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:43:642.413 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:43:642.425 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:43:642.467 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:43:642.500 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:43:642.527 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:43:642.564 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:43:642.894 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:43:642.929 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:43:642.953 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:43:642.987 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:40:43:643.002 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:48:505.754 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:48:505.968 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:48:505.999 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:48:506.049 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:48:506.083 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:48:506.115 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:48:506.152 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:48:506.170 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:48:506.186 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:48:506.199 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:48:642.389 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:48:642.526 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:48:642.572 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:48:642.625 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:48:642.657 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:48:642.698 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:48:642.824 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:48:642.844 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:48:642.891 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:48:642.907 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:48:642.935 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:48:642.985 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:48:643.020 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:48:643.055 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:48:643.083 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:48:643.259 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:48:643.292 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:48:643.316 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:48:643.355 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:40:48:643.383 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:53:506.043 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:53:506.234 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:53:506.265 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:53:506.301 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:53:506.335 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:53:506.366 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:53:506.402 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:53:506.420 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:53:506.436 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:53:506.450 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:53:642.785 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:53:643.006 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:53:643.071 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:53:643.119 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:53:643.218 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:53:643.305 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:53:643.360 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:53:643.387 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:53:643.411 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:53:643.442 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:53:643.460 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:53:643.512 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:53:643.588 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:53:643.640 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:53:643.687 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:53:643.802 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:53:643.854 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:53:643.885 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:53:643.918 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:40:53:643.939 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:53:805.104 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-26 Thu 05:40:53:805.360 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-26 Thu 05:40:53:805.463 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:40:53:805.494 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: SANDESH: Send FAILED: 1774503653805183 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:40:53:805.505 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:40:53:805.516 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:40:53:805.713 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: SANDESH: Send FAILED: 1774503653805449 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 51, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:40:53:805.738 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:53:805.778 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-26 Thu 05:40:53:805.788 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1):EvTcpConnectFail 2026-03-26 Thu 05:40:53:805.834 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-26 Thu 05:40:53:805.845 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Connect : EvTcpConnectFail 2026-03-26 Thu 05:40:53:805.955 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-26 Thu 05:40:53:805.985 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Idle 2026-03-26 Thu 05:40:53:806.051 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:53:806.221 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: SANDESH: Send FAILED: 1774503653805907 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:40:53:806.244 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:53:806.256 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:53:806.280 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: SANDESH: Send FAILED: 1774503653805933 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:40:53:806.289 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:40:53:806.298 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-26 Thu 05:40:53:806.308 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:40:53:806.453 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: SANDESH: Send FAILED: 1774503653806041 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:40:53:806.468 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534916019968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:40:58:506.357 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:40:58:506.662 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:40:58:506.698 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:40:58:506.886 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:40:58:506.943 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:40:58:506.991 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:40:58:507.067 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:40:58:507.097 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:40:58:507.117 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:40:58:507.136 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:40:58:643.395 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:40:58:643.519 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:40:58:643.565 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:40:58:643.621 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:40:58:643.660 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:40:58:643.703 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:40:58:643.751 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:40:58:643.767 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:40:58:643.793 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:40:58:643.807 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:40:58:643.833 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:40:58:643.881 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:40:58:643.923 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:40:58:643.966 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:40:58:643.997 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:40:58:644.177 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:40:58:644.206 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:40:58:644.237 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:40:58:644.252 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:40:58:644.266 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:41:03:506.756 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:41:03:506.939 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:41:03:507.112 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:41:03:507.172 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:41:03:507.189 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:41:03:507.223 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:41:03:507.259 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:41:03:507.297 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:41:03:507.320 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:41:03:507.334 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:41:03:643.798 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:41:03:643.937 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:41:03:643.984 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:41:03:644.041 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:41:03:644.096 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:41:03:644.150 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:41:03:644.187 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:41:03:644.212 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:41:03:644.234 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:41:03:644.257 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:41:03:644.269 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:41:03:644.302 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:41:03:644.337 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:41:03:644.366 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:41:03:644.393 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:41:03:644.584 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:41:03:644.604 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:41:03:644.620 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:41:03:644.651 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:41:03:644.822 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:41:06:939.619 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535454930688, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:41:06:939.730 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535454930688, Pid 1]: SANDESH: Send FAILED: 1774503666939451 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 84 dequeues = 84 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 112 dequeues = 112 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774503653805984 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774503653806307 ] max_sm_queue_count = 5 ] 2026-03-26 Thu 05:41:06:939.756 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535454930688, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:41:06:939.768 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535454930688, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:41:06:939.940 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535454930688, Pid 1]: SANDESH: Send FAILED: 1774503666939583 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:41:06:939.956 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535454930688, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:41:08:507.020 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:41:08:507.242 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12274e0 2026-03-26 Thu 05:41:08:507.281 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:41:08:507.357 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226ba0 2026-03-26 Thu 05:41:08:507.381 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:41:08:507.442 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:41:08:507.491 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:41:08:507.548 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1226260 2026-03-26 Thu 05:41:08:507.588 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1225920 2026-03-26 Thu 05:41:08:507.609 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1224fe0 2026-03-26 Thu 05:41:08:644.230 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:41:08:644.381 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:41:08:644.420 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:41:08:644.452 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:41:08:644.518 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:41:08:644.577 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221870 2026-03-26 Thu 05:41:08:644.612 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1221340 2026-03-26 Thu 05:41:08:644.628 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x121a1d0 2026-03-26 Thu 05:41:08:644.643 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1220ca0 2026-03-26 Thu 05:41:08:644.675 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12205f0 2026-03-26 Thu 05:41:08:644.687 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:41:08:644.719 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:41:08:644.749 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:41:08:644.776 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:41:08:644.850 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:41:08:644.998 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12246a0 2026-03-26 Thu 05:41:08:645.042 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1222ae0 2026-03-26 Thu 05:41:08:645.107 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x12221a0 2026-03-26 Thu 05:41:08:645.125 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223d60 2026-03-26 Thu 05:41:08:645.139 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: -1 0x1223420 2026-03-26 Thu 05:41:13:507.327 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12274e0 2026-03-26 Thu 05:41:13:507.524 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x12274e0 2026-03-26 Thu 05:41:13:507.536 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:507.582 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:507.593 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226ba0 2026-03-26 Thu 05:41:13:507.680 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1226ba0 2026-03-26 Thu 05:41:13:507.689 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:507.701 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:507.710 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1226260 2026-03-26 Thu 05:41:13:507.757 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1225920 2026-03-26 Thu 05:41:13:507.814 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1224fe0 2026-03-26 Thu 05:41:13:507.890 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:507.905 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:507.930 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1226260 2026-03-26 Thu 05:41:13:507.937 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:507.947 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:507.956 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1225920 2026-03-26 Thu 05:41:13:507.963 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:507.971 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:507.980 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1224fe0 2026-03-26 Thu 05:41:13:507.986 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:507.995 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:508.025 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:508.069 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:508.087 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:508.215 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:508.238 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:508.249 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:508.263 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:508.273 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:508.281 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:41:13:508.288 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:508.337 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:508.360 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:508.379 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:508.397 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:508.421 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: SANDESH: Send FAILED: 1774503673508176 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.242:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:41:13:508.438 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:41:13:644.660 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221870 2026-03-26 Thu 05:41:13:644.851 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1221340 2026-03-26 Thu 05:41:13:644.894 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x121a1d0 2026-03-26 Thu 05:41:13:644.957 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1220ca0 2026-03-26 Thu 05:41:13:645.001 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12205f0 2026-03-26 Thu 05:41:13:645.081 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1221870 2026-03-26 Thu 05:41:13:645.104 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:645.131 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:645.143 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1221340 2026-03-26 Thu 05:41:13:645.150 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:645.159 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:645.168 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x121a1d0 2026-03-26 Thu 05:41:13:645.174 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:645.184 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:645.192 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12246a0 2026-03-26 Thu 05:41:13:645.263 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1220ca0 2026-03-26 Thu 05:41:13:645.272 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:645.284 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:645.300 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x12205f0 2026-03-26 Thu 05:41:13:645.307 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:645.317 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:645.348 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1222ae0 2026-03-26 Thu 05:41:13:645.382 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x12221a0 2026-03-26 Thu 05:41:13:645.439 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223d60 2026-03-26 Thu 05:41:13:645.514 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1223420 2026-03-26 Thu 05:41:13:645.610 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:645.626 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:645.658 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:645.670 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:645.685 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:645.696 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:645.742 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x12246a0 2026-03-26 Thu 05:41:13:645.750 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:645.761 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:645.777 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1222ae0 2026-03-26 Thu 05:41:13:645.784 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:645.799 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:645.877 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:645.957 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:645.978 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:645.996 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x12221a0 2026-03-26 Thu 05:41:13:646.001 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:41:13:646.003 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:646.078 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:646.096 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1223420 2026-03-26 Thu 05:41:13:646.100 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: SANDESH: Send FAILED: 1774503673645915 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.242:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:41:13:646.121 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:41:13:646.302 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:646.314 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:646.324 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: RAC_Connect status: 0 0x1223d60 2026-03-26 Thu 05:41:13:646.330 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:41:13:646.353 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-26 Thu 05:41:13:646.402 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:646.414 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:646.429 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:646.439 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:646.481 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:646.501 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:646.663 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:646.680 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:646.720 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:646.737 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:646.747 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:646.762 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:646.772 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:646.786 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:646.796 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:646.810 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-26 Thu 05:41:13:646.820 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-26 Thu 05:41:13:646.870 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:646.888 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:41:13:646.949 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: SANDESH: Send FAILED: 1774503673646846 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:41:13:646.966 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:41:13:646.890 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:647.424 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:13:647.450 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-26 Thu 05:41:23:806.101 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-26 Thu 05:41:23:806.296 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Connect : Start Connect timer 10.0.0.242:8086 2026-03-26 Thu 05:41:23:806.402 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:41:23:806.433 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503683806177 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.242:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:41:23:806.443 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:41:23:806.453 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:41:23:806.639 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503683806389 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.242:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 149, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 61, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:41:23:806.666 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:41:23:807.213 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-26 Thu 05:41:23:807.242 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: -(-1):EvTcpConnectFail 2026-03-26 Thu 05:41:23:807.771 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-26 Thu 05:41:23:807.785 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Connect : EvTcpConnectFail 2026-03-26 Thu 05:41:23:807.923 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-26 Thu 05:41:23:807.958 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Idle 2026-03-26 Thu 05:41:23:808.029 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:41:23:808.192 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: SANDESH: Send FAILED: 1774503683807870 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 149, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:41:23:808.217 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:41:23:808.230 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:41:23:808.254 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: SANDESH: Send FAILED: 1774503683807898 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:41:23:808.263 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:41:23:808.273 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-26 Thu 05:41:23:808.290 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:41:23:808.436 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: SANDESH: Send FAILED: 1774503683808017 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 150, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:41:23:808.452 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:41:36:940.339 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-26 Thu 05:41:36:940.405 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:41:36:940.484 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503696940208 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 94 dequeues = 94 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 125 dequeues = 125 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774503683807956 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774503683808289 ] max_sm_queue_count = 5 ] 2026-03-26 Thu 05:41:36:940.501 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:41:36:940.517 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:41:36:940.694 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: SANDESH: Send FAILED: 1774503696940319 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 150, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:41:36:940.721 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534928615168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:41:42:119.438 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:41:42:119.540 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: SANDESH: Send FAILED: 1774503702119394 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-26 Thu 05:41:42:119.562 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535463327488, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-03-26 Thu 05:41:53:808.668 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-26 Thu 05:41:53:808.868 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-26 Thu 05:41:53:808.975 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:41:53:809.007 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SANDESH: Send FAILED: 1774503713808742 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:41:53:809.018 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:41:53:809.038 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:41:53:809.212 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SANDESH: Send FAILED: 1774503713808963 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 150, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 69, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:41:53:809.232 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:41:53:811.833 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:41556::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-26 Thu 05:41:53:811.861 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:41556::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-03-26 Thu 05:41:53:811.870 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535615395968, Pid 1]: 10.0.0.33:41556::10.0.0.38:8086(-1):EvTcpConnected 2026-03-26 Thu 05:41:53:811.953 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-26 Thu 05:41:53:811.986 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-26 Thu 05:41:53:812.033 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: ClientInit 2026-03-26 Thu 05:41:53:812.056 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5546-3.:contrail-query-engine:0:Database count 4 2026-03-26 Thu 05:41:53:812.186 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:41:53:812.198 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:41:53:812.224 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: SANDESH: Send FAILED: 1774503713812023 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:41:53:812.234 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:41:53:812.244 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:41:53:812.251 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-26 Thu 05:41:53:812.264 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:41:53:812.271 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-26 Thu 05:41:53:812.456 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: SANDESH: Send FAILED: 1774503713812173 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 28, ] ] ] 2026-03-26 Thu 05:41:53:812.482 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534937011968, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-26 Thu 05:41:53:846.847 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: OnMessage control in state: ClientInit 2026-03-26 Thu 05:41:53:846.904 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-26 Thu 05:41:53:846.915 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-26 Thu 05:41:53:846.942 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Received Ctrl Message with size 0 2026-03-26 Thu 05:41:53:849.719 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 193334 calls = 27 average_bytes = 7160 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 94 dequeues = 94 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 125 dequeues = 125 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774503683807956 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774503683808289 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 20 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 157315 calls = 19 average_bytes = 8279 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 28, ] ] max_sm_queue_count = 5 proxy seq 55 2026-03-26 Thu 05:41:53:853.142 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-26 Thu 05:41:53:853.195 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 36 2026-03-26 Thu 05:41:53:853.221 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-26 Thu 05:41:53:853.272 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5546-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 6 2026-03-26 Thu 05:41:53:853.284 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2026-03-26 Thu 05:41:53:853.293 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-26 Thu 05:41:53:853.337 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534920218368, Pid 1]: Established 2026-03-26 Thu 05:42:36:941.947 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-26 Thu 05:43:36:942.680 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534945408768, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-26 Thu 05:44:36:943.329 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535429080832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-26 Thu 05:45:36:947.079 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-03-26 Thu 05:46:36:950.230 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535467525888, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2026-03-26 Thu 05:47:36:951.676 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140535454930688, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2026-03-26 Thu 05:48:03:173.980 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvStop in state Established 2026-03-26 Thu 05:48:03:174.092 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Idle 2026-03-26 Thu 05:48:03:174.212 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-26 Thu 05:48:03:174.225 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-26 Thu 05:48:03:174.406 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: SANDESH: Send FAILED: 1774504083174196 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774503396931278 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 540379 calls = 61 average_bytes = 8858 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 11 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 65, ] ] ] 2026-03-26 Thu 05:48:03:174.434 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 140534941210368, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:48:30:237.603 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: http-server-port 8091 2026-03-26 Thu 05:48:30:237.685 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Max-tasks 16 2026-03-26 Thu 05:48:30:237.694 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Max-slice 100 2026-03-26 Thu 05:48:30:237.700 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Collectors 10.0.0.38:8086 2026-03-26 Thu 05:48:30:237.706 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Collectors 10.0.0.242:8086 2026-03-26 Thu 05:48:30:237.711 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Collectors 10.0.0.33:8086 2026-03-26 Thu 05:48:30:237.737 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: ROLE : Generator 2026-03-26 Thu 05:48:30:237.744 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-26 Thu 05:48:30:237.749 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5546-3. 2026-03-26 Thu 05:48:30:237.755 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-26 Thu 05:48:30:237.761 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-26 Thu 05:48:30:237.766 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-26 Thu 05:48:30:237.773 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-26 Thu 05:48:30:251.785 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: HTTP Introspect Init 2026-03-26 Thu 05:48:30:251.935 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 138 2026-03-26 Thu 05:48:30:252.039 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Sandesh Http Server Port: 8091 2026-03-26 Thu 05:48:30:252.112 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-26 Thu 05:48:30:252.122 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-26 Thu 05:48:30:252.527 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Idle 2026-03-26 Thu 05:48:30:252.690 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: No Client: 1774504110252628 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774504110252540 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:48:30:252.895 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-26 Thu 05:48:30:252.915 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-26 Thu 05:48:30:252.929 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.242:9042 10.0.0.33:9042 2026-03-26 Thu 05:48:30:253.008 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Processing scm::EvStart in state Idle 2026-03-26 Thu 05:48:30:253.097 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Disconnect 2026-03-26 Thu 05:48:30:253.163 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-26 Thu 05:48:30:253.374 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-26 Thu 05:48:30:253.457 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:48:30:253.497 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: SANDESH: Send FAILED: 1774504110253090 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:48:30:253.507 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:48:30:253.517 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:48:30:253.638 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: SANDESH: Send FAILED: 1774504110253151 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774504110252540 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:48:30:253.661 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:48:30:253.675 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:48:30:253.750 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: SANDESH: Send FAILED: 1774504110253197 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774504110252540 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:48:30:253.766 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:48:30:253.775 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:48:30:253.790 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: SANDESH: Send FAILED: 1774504110253216 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:48:30:253.797 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:48:30:253.805 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:48:30:253.862 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: SANDESH: Send FAILED: 1774504110253438 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774504110252540 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:48:30:253.878 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-26 Thu 05:48:30:360.893 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-26 Thu 05:48:30:367.766 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:32972::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2026-03-26 Thu 05:48:30:396.582 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: 10.0.0.33:32972::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-03-26 Thu 05:48:30:396.611 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: 10.0.0.33:32972::10.0.0.38:8086(-1):EvTcpConnected 2026-03-26 Thu 05:48:30:396.664 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d50540 2026-03-26 Thu 05:48:30:396.671 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:396.689 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:396.705 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d55540 2026-03-26 Thu 05:48:30:396.710 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:396.719 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:396.739 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d55df0 2026-03-26 Thu 05:48:30:396.745 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:396.753 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:396.764 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d56ab0 2026-03-26 Thu 05:48:30:396.770 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:396.778 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:396.789 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d57770 2026-03-26 Thu 05:48:30:396.795 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:396.803 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:396.814 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d5c0f0 2026-03-26 Thu 05:48:30:396.819 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:396.827 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:396.839 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d5cd30 2026-03-26 Thu 05:48:30:396.847 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:396.858 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:396.874 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d5d970 2026-03-26 Thu 05:48:30:396.883 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:396.894 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:396.910 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d5e5b0 2026-03-26 Thu 05:48:30:396.918 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:396.929 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:396.939 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d58430 2026-03-26 Thu 05:48:30:396.944 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:396.956 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:396.967 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d58ff0 2026-03-26 Thu 05:48:30:396.972 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:396.980 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:396.990 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d5f1f0 2026-03-26 Thu 05:48:30:396.995 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:397.003 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:397.013 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d59c30 2026-03-26 Thu 05:48:30:397.018 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:397.026 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:397.036 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d5a870 2026-03-26 Thu 05:48:30:397.042 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:397.049 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:397.059 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: RAC_Connect status: 0 0x1d5b4b0 2026-03-26 Thu 05:48:30:397.064 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: Connected to REDIS... 2026-03-26 Thu 05:48:30:397.071 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.242 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-26 Thu 05:48:30:367.835 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-26 Thu 05:48:30:397.313 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: SANDESH: Send FAILED: 1774504110366034 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:48:30:397.322 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-26 Thu 05:48:30:397.337 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-26 Thu 05:48:30:397.349 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-26 Thu 05:48:30:397.392 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: ClientInit 2026-03-26 Thu 05:48:30:397.410 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5546-3.:contrail-query-engine:0:Database count 1 2026-03-26 Thu 05:48:30:397.537 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:48:30:397.548 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:48:30:397.564 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: SANDESH: Send FAILED: 1774504110397384 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-26 Thu 05:48:30:397.572 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:48:30:397.580 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:48:30:397.586 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-26 Thu 05:48:30:397.609 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:48:30:397.616 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-26 Thu 05:48:30:397.733 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: SANDESH: Send FAILED: 1774504110397520 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774504110252540 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-26 Thu 05:48:30:397.754 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-26 Thu 05:48:30:398.458 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.476 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.499 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.515 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.527 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.536 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.547 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.556 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.567 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.575 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.588 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.596 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.608 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.616 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.628 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.636 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.696 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:398.720 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:398.734 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:398.746 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:398.764 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:398.787 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:398.814 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:398.827 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:398.840 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.849 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.861 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.869 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.901 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:398.914 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:398.930 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.938 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.950 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.958 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.969 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:398.978 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:398.989 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:399.001 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:399.012 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-26 Thu 05:48:30:399.021 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.242:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-26 Thu 05:48:30:399.034 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:399.053 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:399.065 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:399.076 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:399.087 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-26 Thu 05:48:30:399.139 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:48:30:399.149 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:48:30:399.167 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: SANDESH: Send FAILED: 1774504110398671 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-26 Thu 05:48:30:399.174 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:48:30:399.182 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:48:30:399.188 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:48:30:399.202 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: SANDESH: Send FAILED: 1774504110398715 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.242:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:48:30:399.212 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:48:30:399.220 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:48:30:399.226 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:48:30:399.239 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: SANDESH: Send FAILED: 1774504110398806 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.242:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:48:30:399.245 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:48:30:399.252 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-26 Thu 05:48:30:399.258 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-26 Thu 05:48:30:399.271 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: SANDESH: Send FAILED: 1774504110399049 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-26 Thu 05:48:30:399.277 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-26 Thu 05:48:30:401.297 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897751619136, Pid 1]: OnMessage control in state: ClientInit 2026-03-26 Thu 05:48:30:401.331 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897751619136, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-26 Thu 05:48:30:401.340 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897751619136, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-26 Thu 05:48:30:401.362 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897751619136, Pid 1]: Received Ctrl Message with size 0 2026-03-26 Thu 05:48:30:401.484 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897751619136, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5546-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774504110252540 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, (*_iter40) = 10.0.0.242:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2026-03-26 Thu 05:48:30:401.516 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897751619136, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-26 Thu 05:48:30:401.538 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897751619136, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5546-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.242:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.242:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2026-03-26 Thu 05:48:30:401.550 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897751619136, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-26 Thu 05:48:30:401.557 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897751619136, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-26 Thu 05:48:30:401.564 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897751619136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-26 Thu 05:48:30:401.595 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897751619136, Pid 1]: Established 2026-03-26 Thu 05:49:30:253.211 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-26 Thu 05:50:30:254.073 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897692841536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-26 Thu 05:51:30:254.725 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897726428736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-26 Thu 05:52:30:255.348 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897697039936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-26 Thu 05:53:30:256.041 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-26 Thu 05:54:30:256.815 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-26 Thu 05:55:30:257.452 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897734825536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-26 Thu 05:56:30:258.060 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897697039936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-26 Thu 05:57:30:258.775 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-03-26 Thu 05:58:30:259.391 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897747420736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2026-03-26 Thu 05:59:30:260.628 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2026-03-26 Thu 06:00:30:261.850 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897747420736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2026-03-26 Thu 06:01:30:262.504 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897734825536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2026-03-26 Thu 06:02:30:263.206 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897747420736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2026-03-26 Thu 06:03:30:263.913 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897697039936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2026-03-26 Thu 06:04:30:264.477 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897747420736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2026-03-26 Thu 06:04:53:040.192 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.33:47424 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-26 Thu 06:05:08:476.620 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.33:49402 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-26 Thu 06:05:30:267.080 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2026-03-26 Thu 06:06:30:268.453 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897692841536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2026-03-26 Thu 06:06:44:972.694 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.38:41758 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-26 Thu 06:07:09:842.085 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.38 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-03-26 Thu 06:07:09:842.751 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 06456231-28da-11f1-aed0-00000a000026 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-03-26 Thu 06:07:09:842.803 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QEQueryLog: 06456231-28da-11f1-aed0-00000a000026 :0:json query is: { end_time: 1774505229825948 , enqueue_time: 1774505229842159 , query_metadata: {"enqueue_time": 1774505229841971} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774505209384420 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-11157719:ctest-vn22-93015443:ctest-vn22-93015443", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-26 Thu 06:07:09:842.983 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-03-26 Thu 06:07:09:843.000 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-03-26 Thu 06:07:09:843.866 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897755817536, Pid 1]: [SYS_INFO]: QEQueryLog: 06456231-28da-11f1-aed0-00000a000026 :0:json query is: { end_time: 1774505229825948 , enqueue_time: 1774505229842159 , query_metadata: {"enqueue_time": 1774505229841971} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774505209384420 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-11157719:ctest-vn22-93015443:ctest-vn22-93015443", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-26 Thu 06:07:09:844.445 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 06456231-28da-11f1-aed0-00000a000026 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-03-26 Thu 06:07:09:895.435 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:895.511 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:895.526 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:895.539 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:895.552 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:895.564 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:895.592 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:895.604 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:895.615 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:895.628 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:902.136 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:902.182 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:902.191 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:902.199 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:902.206 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:902.214 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:902.221 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:902.228 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:902.236 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:902.243 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:902.286 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:902.322 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:902.351 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:902.359 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:902.367 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:902.374 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:902.381 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:902.388 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:902.395 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 UNION between tables of sizes 0 and 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-26 Thu 06:07:09:902.415 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: [SYS_INFO]: QELog: 06456231-28da-11f1-aed0-00000a000026 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-26 Thu 06:07:09:902.671 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: [SYS_INFO]: QEQueryLog: 06456231-28da-11f1-aed0-00000a000026 ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-11157719:ctest-vn22-93015443:ctest-vn22-93015443 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-26 Thu 06:07:09:902.336 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QEQueryLog: 06456231-28da-11f1-aed0-00000a000026 :0:json query is: { end_time: 1774505229825948 , enqueue_time: 1774505229842159 , query_metadata: {"enqueue_time": 1774505229841971} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774505209384420 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-11157719:ctest-vn22-93015443:ctest-vn22-93015443", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-26 Thu 06:07:09:903.007 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-03-26 Thu 06:07:09:903.041 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-03-26 Thu 06:07:09:903.084 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-03-26 Thu 06:07:09:903.456 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: Finished: QID 06456231-28da-11f1-aed0-00000a000026 Table ObjectRoutingInstance Time(ms) 60 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-03-26 Thu 06:07:09:903.523 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-03-26 Thu 06:07:09:903.566 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5546-3. table = ObjectRoutingInstance query_stats= [ qid = 06456231-28da-11f1-aed0-00000a000026 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-11157719:ctest-vn22-93015443:ctest-vn22-93015443", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 57,0, 51,0, 57,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 60 rows = 1 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-03-26 Thu 06:07:30:269.614 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897739023936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2026-03-26 Thu 06:08:30:270.203 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897734825536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2026-03-26 Thu 06:09:30:270.916 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2026-03-26 Thu 06:10:30:271.499 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897747420736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2026-03-26 Thu 06:11:30:272.846 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2026-03-26 Thu 06:12:30:273.385 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897697039936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2026-03-26 Thu 06:13:30:274.181 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897747420736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2026-03-26 Thu 06:14:30:274.758 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897697039936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2026-03-26 Thu 06:15:30:275.267 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897722230336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2026-03-26 Thu 06:16:30:275.785 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897718031936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2026-03-26 Thu 06:17:30:276.380 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897734825536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2026-03-26 Thu 06:18:30:279.025 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897747420736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2026-03-26 Thu 06:19:30:279.612 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897705436736, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2026-03-26 Thu 06:19:37:697.889 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.38:53940 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-26 Thu 06:20:30:280.150 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897730627136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2026-03-26 Thu 06:21:30:280.780 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897713833536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2026-03-26 Thu 06:21:55:269.433 UTC cn-jenkins-deploy-platform-ansible-os-5546-3 [Thread 139897762957440, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.33:8091::10.0.0.33:33792 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256