2026-03-23 Mon 05:37:10:985.602 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: http-server-port 8091 2026-03-23 Mon 05:37:10:985.680 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Max-tasks 16 2026-03-23 Mon 05:37:10:985.688 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Max-slice 100 2026-03-23 Mon 05:37:10:985.694 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Collectors 10.0.0.38:8086 2026-03-23 Mon 05:37:10:985.699 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Collectors 10.0.0.50:8086 2026-03-23 Mon 05:37:10:985.704 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Collectors 10.0.0.33:8086 2026-03-23 Mon 05:37:10:985.721 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: ROLE : Generator 2026-03-23 Mon 05:37:10:985.727 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-23 Mon 05:37:10:985.732 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5515-1. 2026-03-23 Mon 05:37:10:985.737 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-23 Mon 05:37:10:985.742 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-23 Mon 05:37:10:985.747 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-23 Mon 05:37:10:985.753 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-23 Mon 05:37:10:986.321 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: HTTP Introspect Init 2026-03-23 Mon 05:37:10:986.417 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2026-03-23 Mon 05:37:10:986.472 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Sandesh Http Server Port: 8091 2026-03-23 Mon 05:37:10:986.509 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-23 Mon 05:37:10:986.517 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-23 Mon 05:37:10:986.588 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Idle 2026-03-23 Mon 05:37:10:986.736 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: No Client: 1774244230986665 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:10:986.850 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-23 Mon 05:37:10:986.863 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-23 Mon 05:37:10:986.875 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-03-23 Mon 05:37:10:988.104 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvStart in state Idle 2026-03-23 Mon 05:37:10:988.168 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Disconnect 2026-03-23 Mon 05:37:10:988.238 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-23 Mon 05:37:10:988.438 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:37:10:988.505 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:988.530 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SANDESH: Send FAILED: 1774244230988160 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:10:988.549 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:10:988.561 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:988.645 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SANDESH: Send FAILED: 1774244230988227 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:10:988.663 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:10:988.672 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:988.740 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SANDESH: Send FAILED: 1774244230988285 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:10:988.766 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:10:988.775 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:988.791 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SANDESH: Send FAILED: 1774244230988305 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:10:988.797 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:10:988.805 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:988.899 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SANDESH: Send FAILED: 1774244230988494 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:10:988.914 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:10:996.908 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:10:996.937 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:10:996.997 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:10:997.044 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:10:997.086 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: SANDESH: Send FAILED: 1774244230996971 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:10:997.095 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:16:015.162 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:16:015.207 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:16:015.248 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:21:017.526 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:21:017.562 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:21:017.583 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:26:022.403 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2026-03-23 Mon 05:37:26:022.513 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:26:022.536 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:31:250.167 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-23 Mon 05:37:31:250.221 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:31:250.245 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:36:282.682 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2026-03-23 Mon 05:37:36:282.725 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:36:282.748 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:41:359.814 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2026-03-23 Mon 05:37:41:359.868 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2026-03-23 Mon 05:37:46:390.679 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-23 Mon 05:37:51:395.876 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:51:395.965 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: SANDESH: Send FAILED: 1774244271395796 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:51:395.977 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:395.925 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:37:51:396.414 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:37:51:396.441 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27eb5f0 2026-03-23 Mon 05:37:51:396.447 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.461 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.474 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ebca0 2026-03-23 Mon 05:37:51:396.479 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.487 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.497 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27e51d0 2026-03-23 Mon 05:37:51:396.502 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.510 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.519 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ec340 2026-03-23 Mon 05:37:51:396.524 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.532 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.541 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ec870 2026-03-23 Mon 05:37:51:396.546 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.554 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.562 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ed1a0 2026-03-23 Mon 05:37:51:396.567 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.583 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.591 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27edae0 2026-03-23 Mon 05:37:51:396.596 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.604 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.612 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ee420 2026-03-23 Mon 05:37:51:396.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.624 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.633 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27effe0 2026-03-23 Mon 05:37:51:396.637 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.645 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.654 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ef6a0 2026-03-23 Mon 05:37:51:396.658 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.666 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.674 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f1260 2026-03-23 Mon 05:37:51:396.679 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.687 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.695 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27eed60 2026-03-23 Mon 05:37:51:396.700 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.707 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.715 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f0920 2026-03-23 Mon 05:37:51:396.720 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.730 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.739 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f1ba0 2026-03-23 Mon 05:37:51:396.747 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.754 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.763 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f24e0 2026-03-23 Mon 05:37:51:396.768 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:37:51:396.775 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:37:51:396.997 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:37:51:397.013 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:37:51:397.149 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:37:51:397.191 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Idle 2026-03-23 Mon 05:37:51:397.304 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:397.564 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SANDESH: Send FAILED: 1774244271397103 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 36 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 37, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:51:397.586 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:397.597 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:397.618 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SANDESH: Send FAILED: 1774244271397127 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:37:51:397.625 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:397.634 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:37:51:397.676 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:397.777 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SANDESH: Send FAILED: 1774244271397275 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 36 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 38, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:51:397.794 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:397.871 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:397.914 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SANDESH: Send FAILED: 1774244271397810 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244271397189 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244271397674 ] max_sm_queue_count = 5 ] 2026-03-23 Mon 05:37:51:397.922 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:397.930 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:398.025 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SANDESH: Send FAILED: 1774244271397859 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 36 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 38, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:51:398.039 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:51:398.570 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.586 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.601 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.610 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.623 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.633 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.651 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.660 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.672 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.682 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.697 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.706 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.718 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.727 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.739 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.748 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.760 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.769 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.780 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:398.789 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:398.836 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.865 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.882 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.902 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.917 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.940 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.966 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.982 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:398.997 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:399.012 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:399.256 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:399.293 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: SANDESH: Send FAILED: 1774244271398823 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:37:51:399.302 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:399.311 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:399.341 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: SANDESH: Send FAILED: 1774244271398859 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:51:399.349 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:399.369 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:399.387 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: SANDESH: Send FAILED: 1774244271398958 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:51:399.394 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:51:400.622 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:400.649 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:400.665 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:400.674 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:400.688 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:400.697 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:400.709 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:400.718 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:400.730 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:37:51:400.739 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:37:51:400.759 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:400.796 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:400.818 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:400.831 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:400.844 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:37:51:400.920 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:37:51:400.952 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SANDESH: Send FAILED: 1774244271400781 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:51:400.961 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:52:397.284 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:37:52:397.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:37:52:397.592 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:52:397.622 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SANDESH: Send FAILED: 1774244272397357 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:52:397.631 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:37:52:397.639 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:37:52:397.784 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SANDESH: Send FAILED: 1774244272397582 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:52:397.808 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:37:52:400.168 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:45434::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:37:52:400.191 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:45434::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:37:52:400.199 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:45434::10.0.0.50:8086(-1):EvTcpConnected 2026-03-23 Mon 05:37:52:400.261 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:37:52:400.298 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:37:52:400.348 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: ClientInit 2026-03-23 Mon 05:37:52:400.377 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-1.:contrail-query-engine:0:Database count 1 2026-03-23 Mon 05:37:52:400.471 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:37:52:400.481 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:37:52:400.503 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SANDESH: Send FAILED: 1774244272400338 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:37:52:400.511 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:37:52:400.520 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:37:52:400.526 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:37:52:400.538 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:37:52:400.544 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:37:52:400.645 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SANDESH: Send FAILED: 1774244272400460 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:37:52:400.669 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:37:52:408.995 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:37:52:409.062 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:37:52:409.072 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:37:52:409.096 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:37:52:409.266 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244271397189 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244271397674 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 81 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2026-03-23 Mon 05:37:52:409.296 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:37:52:409.322 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2026-03-23 Mon 05:37:52:409.334 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:37:52:409.344 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:37:52:409.352 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:37:52:409.383 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Established 2026-03-23 Mon 05:38:00:037.647 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: OnMessage control in state: Established 2026-03-23 Mon 05:38:00:037.726 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2026-03-23 Mon 05:38:00:037.737 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: Established : EvSandeshMessageRecv 2026-03-23 Mon 05:38:00:037.779 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: Received Ctrl Message with size 3 2026-03-23 Mon 05:38:00:037.792 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2026-03-23 Mon 05:38:00:037.802 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2026-03-23 Mon 05:38:00:037.815 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2026-03-23 Mon 05:38:00:037.822 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:38:21:398.142 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-23 Mon 05:39:21:399.355 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-23 Mon 05:40:21:399.864 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-23 Mon 05:40:56:616.783 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:39554 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-23 Mon 05:41:18:814.742 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:33338 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2026-03-23 Mon 05:41:21:400.420 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-23 Mon 05:42:21:401.975 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897864177408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-23 Mon 05:43:14:841.645 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27eed60 2026-03-23 Mon 05:43:14:841.708 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.839 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27ef6a0 2026-03-23 Mon 05:43:14:841.857 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.901 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27edae0 2026-03-23 Mon 05:43:14:841.913 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.933 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27ee420 2026-03-23 Mon 05:43:14:841.943 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:841.976 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:43:14:841.983 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27ed1a0 2026-03-23 Mon 05:43:14:841.991 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:848.787 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:43:14:849.730 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:43:14:849.761 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:43:14:849.780 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:43:14:849.798 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:43:14:876.240 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27f24e0 2026-03-23 Mon 05:43:14:876.293 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:876.340 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27f0920 2026-03-23 Mon 05:43:14:876.370 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:876.506 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27f1ba0 2026-03-23 Mon 05:43:14:876.520 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:876.552 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:43:14:876.560 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27effe0 2026-03-23 Mon 05:43:14:876.568 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:876.593 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27f1260 2026-03-23 Mon 05:43:14:876.602 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:43:14:877.700 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:43:14:877.757 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:43:14:877.794 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:43:14:877.815 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:43:14:877.867 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:43:19:849.719 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:43:19:849.843 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:43:19:849.877 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:43:19:849.903 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:43:19:849.930 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:43:19:850.120 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:43:19:850.152 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:43:19:850.172 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:43:19:850.185 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:43:19:850.197 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:43:19:877.762 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:43:19:877.883 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:43:19:877.917 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:43:19:877.944 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:43:19:877.981 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:43:19:878.184 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:43:19:878.223 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:43:19:878.253 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:43:19:878.266 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:43:19:878.278 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:43:21:407.115 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-23 Mon 05:43:24:750.708 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.38:45434::10.0.0.50:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2026-03-23 Mon 05:43:24:750.824 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.38:45434::10.0.0.50:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:43:24:750.863 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:45434::10.0.0.50:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-23 Mon 05:43:24:750.955 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:45434::10.0.0.50:8086(-1):EvTcpClose 2026-03-23 Mon 05:43:24:751.748 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvTcpClose in state Established 2026-03-23 Mon 05:43:24:751.780 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Established : EvTcpClose 2026-03-23 Mon 05:43:24:754.495 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-23 Mon 05:43:24:754.676 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:24:754.921 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: SANDESH: Send FAILED: 1774244604751886 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = cn-jenkins-deploy-platform-ansible-os-5515-2. collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 311702 calls = 35 average_bytes = 8905 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 93 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 24 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 100, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 36, ] ] ] 2026-03-23 Mon 05:43:24:754.970 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:24:754.988 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:24:755.024 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: SANDESH: Send FAILED: 1774244604751928 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:43:24:755.035 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:43:24:755.050 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2026-03-23 Mon 05:43:24:755.064 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:24:755.310 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: SANDESH: Send FAILED: 1774244604754633 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 311702 calls = 35 average_bytes = 8905 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 93 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 24 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 100, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 36, ] ] ] 2026-03-23 Mon 05:43:24:755.334 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:24:761.225 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:56390::10.0.0.33:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:43:24:761.254 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:56390::10.0.0.33:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:43:24:761.266 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:56390::10.0.0.33:8086(-1):EvTcpConnected 2026-03-23 Mon 05:43:24:762.436 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:43:24:762.463 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:43:24:762.514 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: ClientInit 2026-03-23 Mon 05:43:24:762.536 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-1.:contrail-query-engine:0:Database count 2 2026-03-23 Mon 05:43:24:762.647 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:24:762.661 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:43:24:762.692 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: SANDESH: Send FAILED: 1774244604762503 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:43:24:762.703 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:43:24:762.716 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:24:762.726 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:43:24:762.745 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:24:762.754 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:43:24:762.978 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: SANDESH: Send FAILED: 1774244604762633 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 311702 calls = 35 average_bytes = 8905 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 93 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 24 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 101, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 36, ] ] ] 2026-03-23 Mon 05:43:24:763.003 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:43:24:850.308 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:43:24:850.455 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:43:24:850.528 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:43:24:850.576 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:43:24:850.620 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:43:24:850.826 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:43:24:850.883 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:43:24:850.905 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:43:24:850.924 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:43:24:850.956 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:43:24:878.262 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:43:24:878.383 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:43:24:878.418 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:43:24:878.446 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:43:24:878.473 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:43:24:878.657 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:43:24:878.690 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:43:24:878.709 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:43:24:878.721 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:43:24:878.734 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:43:24:928.174 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.38:56390::10.0.0.33:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2026-03-23 Mon 05:43:24:928.219 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.38:56390::10.0.0.33:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:43:24:928.243 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:56390::10.0.0.33:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-23 Mon 05:43:24:928.255 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:56390::10.0.0.33:8086(-1):EvTcpClose 2026-03-23 Mon 05:43:24:961.731 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-23 Mon 05:43:24:961.774 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: ClientInit : EvTcpClose 2026-03-23 Mon 05:43:24:961.992 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Return to idle with 1000 2026-03-23 Mon 05:43:24:962.008 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Idle 2026-03-23 Mon 05:43:24:962.095 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:43:24:962.341 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: SANDESH: Send FAILED: 1774244604961916 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 313275 calls = 36 average_bytes = 8702 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 93 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 24 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 103, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 37, ] ] ] 2026-03-23 Mon 05:43:24:962.380 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:24:962.395 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:43:24:962.456 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: SANDESH: Send FAILED: 1774244604961964 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:43:24:962.466 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:43:24:962.478 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:43:24:962.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:43:24:962.838 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: SANDESH: Send FAILED: 1774244604962082 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 313275 calls = 36 average_bytes = 8702 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 93 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 24 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 103, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 37, ] ] ] 2026-03-23 Mon 05:43:24:962.876 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:25:962.477 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:43:25:962.686 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:43:25:962.802 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:25:962.832 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: SANDESH: Send FAILED: 1774244605962543 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:43:25:962.847 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:43:25:962.856 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:43:25:963.014 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: SANDESH: Send FAILED: 1774244605962788 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 313275 calls = 36 average_bytes = 8702 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 93 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 24 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 103, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 37, ] ] ] 2026-03-23 Mon 05:43:25:963.029 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:43:25:967.036 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:58326::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:43:25:967.058 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:58326::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:43:25:967.066 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:58326::10.0.0.38:8086(-1):EvTcpConnected 2026-03-23 Mon 05:43:25:970.238 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:43:25:970.262 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:43:25:970.306 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: ClientInit 2026-03-23 Mon 05:43:25:970.329 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-1.:contrail-query-engine:0:Database count 3 2026-03-23 Mon 05:43:25:970.411 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:25:970.420 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:43:25:970.442 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SANDESH: Send FAILED: 1774244605970298 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:43:25:970.449 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:43:25:970.457 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:25:970.464 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:43:25:970.474 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:43:25:970.480 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:43:25:970.620 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SANDESH: Send FAILED: 1774244605970401 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 313275 calls = 36 average_bytes = 8702 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 93 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 24 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 104, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 37, ] ] ] 2026-03-23 Mon 05:43:25:970.638 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:43:25:975.093 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:43:25:975.151 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:43:25:975.162 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:43:25:975.185 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:43:25:975.435 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 313275 calls = 36 average_bytes = 8702 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 54 dequeues = 54 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 62 dequeues = 62 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Established state_since = 1774244272409382 last_state = Established last_event = scm::EvSandeshSend last_event_at = 1774244594876923 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 34 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 289081 calls = 33 average_bytes = 8760 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 93 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 24 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 104, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 37, ] ] max_sm_queue_count = 5 proxy seq 40 2026-03-23 Mon 05:43:25:975.467 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:43:25:975.490 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 20 2026-03-23 Mon 05:43:25:975.501 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:43:25:975.532 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 6 2026-03-23 Mon 05:43:25:975.541 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:43:25:975.548 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:43:25:975.582 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Established 2026-03-23 Mon 05:43:29:850.918 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:43:29:851.051 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:43:29:851.095 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:43:29:851.125 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:43:29:851.156 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:43:29:851.369 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ef6a0 2026-03-23 Mon 05:43:29:851.396 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:29:851.422 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:29:851.439 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27eed60 2026-03-23 Mon 05:43:29:851.447 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:29:851.458 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:29:851.482 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ee420 2026-03-23 Mon 05:43:29:851.488 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:29:851.496 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:29:851.505 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ed1a0 2026-03-23 Mon 05:43:29:851.510 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:29:851.517 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:29:851.526 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27edae0 2026-03-23 Mon 05:43:29:851.533 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:29:851.545 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:29:851.807 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:29:851.826 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:29:851.867 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:29:851.881 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:29:851.890 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:29:851.902 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:29:851.912 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:29:851.924 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:29:851.933 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:29:851.955 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:29:851.966 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:29:852.029 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:29:852.047 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:29:852.062 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:29:852.107 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:29:878.717 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:43:29:878.810 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:43:29:878.844 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:43:29:878.873 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:43:29:878.901 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:43:29:879.037 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:43:29:879.110 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:43:29:879.127 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:43:29:879.140 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:43:29:879.152 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:43:34:882.018 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:43:34:882.125 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:43:34:882.167 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:43:34:882.194 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:43:34:882.223 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:43:34:882.504 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f0920 2026-03-23 Mon 05:43:34:882.519 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:34:882.542 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:34:882.555 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f24e0 2026-03-23 Mon 05:43:34:882.561 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:34:882.570 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:34:882.578 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f1ba0 2026-03-23 Mon 05:43:34:882.583 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:34:882.591 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:34:882.599 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27effe0 2026-03-23 Mon 05:43:34:882.604 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:34:882.612 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:34:882.620 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f1260 2026-03-23 Mon 05:43:34:882.625 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:43:34:882.633 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:43:34:882.863 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:34:882.921 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:34:882.957 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:34:882.973 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:34:882.983 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:34:882.996 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:34:883.005 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:34:883.017 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:34:883.026 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:34:883.039 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:43:34:883.048 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:43:34:887.354 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:34:887.399 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:34:887.414 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:43:34:887.429 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:44:21:409.875 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-23 Mon 05:45:21:411.251 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898238179072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-23 Mon 05:45:55:846.808 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27edae0 2026-03-23 Mon 05:45:55:846.870 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.915 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:45:55:846.924 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27ed1a0 2026-03-23 Mon 05:45:55:846.944 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:846.964 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27ee420 2026-03-23 Mon 05:45:55:846.974 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:847.143 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27ef6a0 2026-03-23 Mon 05:45:55:847.157 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:847.178 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27eed60 2026-03-23 Mon 05:45:55:847.187 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:847.350 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:45:55:847.375 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:45:55:847.388 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:45:55:847.496 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:45:55:847.516 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:45:55:886.283 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27ec870 2026-03-23 Mon 05:45:55:886.343 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.486 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27ec340 2026-03-23 Mon 05:45:55:886.503 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.536 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27e51d0 2026-03-23 Mon 05:45:55:886.547 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.584 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27ebca0 2026-03-23 Mon 05:45:55:886.595 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:886.623 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:45:55:886.631 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27eb5f0 2026-03-23 Mon 05:45:55:886.639 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:895.981 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:45:55:896.077 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.38:58326::10.0.0.38:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:45:55:896.120 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:58326::10.0.0.38:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-23 Mon 05:45:55:896.128 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:58326::10.0.0.38:8086(-1):EvTcpClose 2026-03-23 Mon 05:45:55:896.618 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvTcpClose in state Established 2026-03-23 Mon 05:45:55:896.636 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Established : EvTcpClose 2026-03-23 Mon 05:45:55:896.896 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:45:55:897.021 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:55:897.206 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SANDESH: Send FAILED: 1774244755896730 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = cn-jenkins-deploy-platform-ansible-os-5515-1. collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 492538 calls = 56 average_bytes = 8795 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 145 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 157, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 60, ] ] ] 2026-03-23 Mon 05:45:55:897.229 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:55:897.240 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:55:897.263 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SANDESH: Send FAILED: 1774244755896757 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.50:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:45:55:897.270 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:45:55:897.279 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2026-03-23 Mon 05:45:55:897.288 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:55:897.428 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SANDESH: Send FAILED: 1774244755897008 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 492538 calls = 56 average_bytes = 8795 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 145 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 157, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 60, ] ] ] 2026-03-23 Mon 05:45:55:897.445 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:55:897.712 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:45:55:897.734 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:45:55:897.748 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:45:55:897.762 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:45:55:899.837 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:52938::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:45:55:899.858 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:52938::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:45:55:899.865 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:52938::10.0.0.50:8086(-1):EvTcpConnected 2026-03-23 Mon 05:45:55:904.452 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:45:55:904.471 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:45:55:904.507 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: ClientInit 2026-03-23 Mon 05:45:55:904.520 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-1.:contrail-query-engine:0:Database count 4 2026-03-23 Mon 05:45:55:904.607 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:55:904.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:45:55:904.641 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: SANDESH: Send FAILED: 1774244755904499 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.50:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:45:55:904.655 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:45:55:904.664 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:55:904.671 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:45:55:904.682 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:55:904.689 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:45:55:904.832 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: SANDESH: Send FAILED: 1774244755904596 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 492538 calls = 56 average_bytes = 8795 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 145 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 158, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 60, ] ] ] 2026-03-23 Mon 05:45:55:904.846 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:45:55:957.335 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27f1260 2026-03-23 Mon 05:45:55:957.373 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:957.421 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2026-03-23 Mon 05:45:55:957.430 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27effe0 2026-03-23 Mon 05:45:55:957.439 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:957.458 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27f1ba0 2026-03-23 Mon 05:45:55:957.470 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:957.488 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27f24e0 2026-03-23 Mon 05:45:55:957.500 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:957.518 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Disconnect status: -1 0x27f0920 2026-03-23 Mon 05:45:55:957.528 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2026-03-23 Mon 05:45:55:959.420 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:55:959.436 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:45:55:959.461 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: SANDESH: Send FAILED: 1774244755957579 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:55:959.469 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:45:55:960.284 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:45:55:960.316 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:45:55:960.341 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:45:55:960.541 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:45:55:960.556 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:45:55:970.702 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:52938::10.0.0.50:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-23 Mon 05:45:55:970.729 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:52938::10.0.0.50:8086(-1):EvTcpClose 2026-03-23 Mon 05:45:55:972.775 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-23 Mon 05:45:55:972.795 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: ClientInit : EvTcpClose 2026-03-23 Mon 05:45:55:972.919 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Return to idle with 1000 2026-03-23 Mon 05:45:55:972.939 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Idle 2026-03-23 Mon 05:45:55:973.008 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:55:973.161 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: SANDESH: Send FAILED: 1774244755972871 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 494111 calls = 57 average_bytes = 8668 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 164, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 61, ] ] ] 2026-03-23 Mon 05:45:55:973.180 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:55:973.194 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:55:973.217 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: SANDESH: Send FAILED: 1774244755972900 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:55:973.223 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:45:55:973.232 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:45:55:973.241 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:55:973.373 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: SANDESH: Send FAILED: 1774244755972998 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 494111 calls = 57 average_bytes = 8668 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 164, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 61, ] ] ] 2026-03-23 Mon 05:45:55:973.391 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:56:406.097 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:56:406.183 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: SANDESH: Send FAILED: 1774244756406051 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-23 Mon 05:45:56:406.194 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897851582208, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-03-23 Mon 05:45:56:973.227 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:45:56:973.436 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-23 Mon 05:45:56:973.554 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:56:973.584 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244756973297 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:56:973.593 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:45:56:973.602 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:56:973.747 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244756973537 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 494111 calls = 57 average_bytes = 8668 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 164, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 42, _iter59->first = sent _iter59->second = 61, ] ] ] 2026-03-23 Mon 05:45:56:973.771 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:56:975.823 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:43422::10.0.0.33:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:45:56:975.845 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:43422::10.0.0.33:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:45:56:975.853 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:43422::10.0.0.33:8086(-1):EvTcpConnected 2026-03-23 Mon 05:45:56:976.829 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:45:56:976.847 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:45:56:976.880 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: ClientInit 2026-03-23 Mon 05:45:56:976.895 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-1.:contrail-query-engine:0:Database count 5 2026-03-23 Mon 05:45:56:976.984 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:56:976.993 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:45:56:977.016 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244756976873 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:56:977.033 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:45:56:977.042 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:56:977.049 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:45:56:977.058 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:56:977.065 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:45:56:977.261 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244756976974 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 494111 calls = 57 average_bytes = 8668 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 165, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 44, _iter59->first = sent _iter59->second = 61, ] ] ] 2026-03-23 Mon 05:45:56:977.280 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:45:58:977.023 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: 10.0.0.33:8086 EvConnectTimerExpired in state ClientInit 2026-03-23 Mon 05:45:58:977.088 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2026-03-23 Mon 05:45:58:977.111 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: ClientInit : EvConnectTimerExpired 2026-03-23 Mon 05:45:58:977.300 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Return to idle with 1000 2026-03-23 Mon 05:45:58:977.313 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Idle 2026-03-23 Mon 05:45:58:977.378 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:58:977.543 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: SANDESH: Send FAILED: 1774244758977190 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 495684 calls = 58 average_bytes = 8546 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 165, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 46, _iter59->first = sent _iter59->second = 62, ] ] ] 2026-03-23 Mon 05:45:58:977.563 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:58:977.574 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:58:977.596 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: SANDESH: Send FAILED: 1774244758977224 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:58:977.606 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:45:58:977.615 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:45:58:977.641 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:45:58:977.780 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: SANDESH: Send FAILED: 1774244758977367 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 495684 calls = 58 average_bytes = 8546 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 165, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 46, _iter59->first = sent _iter59->second = 62, ] ] ] 2026-03-23 Mon 05:45:58:977.795 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:59:977.450 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:45:59:977.700 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:45:59:977.843 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:59:977.888 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: SANDESH: Send FAILED: 1774244759977543 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:59:977.904 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:45:59:977.914 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:45:59:978.070 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: SANDESH: Send FAILED: 1774244759977823 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 495684 calls = 58 average_bytes = 8546 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 165, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 49, _iter59->first = sent _iter59->second = 62, ] ] ] 2026-03-23 Mon 05:45:59:978.087 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:45:59:979.654 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:33602::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:45:59:979.684 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:33602::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:45:59:979.692 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:33602::10.0.0.38:8086(-1):EvTcpConnected 2026-03-23 Mon 05:45:59:979.742 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:45:59:979.763 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:45:59:979.797 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: ClientInit 2026-03-23 Mon 05:45:59:979.912 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-1.:contrail-query-engine:0:Database count 6 2026-03-23 Mon 05:45:59:980.018 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:59:980.029 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:45:59:980.051 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SANDESH: Send FAILED: 1774244759979790 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:45:59:980.058 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:45:59:980.067 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:59:980.074 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:45:59:980.103 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:45:59:980.110 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:45:59:980.258 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SANDESH: Send FAILED: 1774244759980004 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 495684 calls = 58 average_bytes = 8546 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 166, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 51, _iter59->first = sent _iter59->second = 62, ] ] ] 2026-03-23 Mon 05:45:59:980.278 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:46:00:847.417 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:00:847.580 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:00:847.631 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:00:847.671 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:00:847.703 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:00:847.820 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:00:847.852 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:00:847.867 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:00:847.938 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:00:847.984 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:00:896.052 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:00:896.259 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:00:897.756 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:00:897.855 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:00:897.909 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:00:897.962 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:00:898.000 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:00:898.036 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:00:898.052 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:00:898.065 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:00:960.353 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:00:960.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:00:960.554 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:00:960.596 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:00:960.635 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:00:960.805 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:00:960.835 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:00:960.863 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:00:960.876 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:00:960.889 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:03:979.907 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: 10.0.0.38:8086 EvConnectTimerExpired in state ClientInit 2026-03-23 Mon 05:46:03:979.967 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2026-03-23 Mon 05:46:03:979.978 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: ClientInit : EvConnectTimerExpired 2026-03-23 Mon 05:46:03:980.155 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Return to idle with 1000 2026-03-23 Mon 05:46:03:980.168 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Idle 2026-03-23 Mon 05:46:03:980.236 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:03:980.386 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244763980054 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 497257 calls = 59 average_bytes = 8428 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 166, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 63, ] ] ] 2026-03-23 Mon 05:46:03:980.419 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:03:980.430 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:03:980.452 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244763980100 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:03:980.460 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:03:980.468 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:03:980.477 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:03:980.621 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244763980220 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 497257 calls = 59 average_bytes = 8428 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 166, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 53, _iter59->first = sent _iter59->second = 63, ] ] ] 2026-03-23 Mon 05:46:03:980.639 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:04:980.319 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:04:980.532 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:46:04:980.628 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:04:980.657 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: SANDESH: Send FAILED: 1774244764980397 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:04:980.665 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:04:980.674 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:04:980.861 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: SANDESH: Send FAILED: 1774244764980617 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 497257 calls = 59 average_bytes = 8428 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 166, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 63, ] ] ] 2026-03-23 Mon 05:46:04:980.879 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:04:982.847 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:57516::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:46:04:982.867 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:57516::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:46:04:982.875 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:57516::10.0.0.50:8086(-1):EvTcpConnected 2026-03-23 Mon 05:46:04:982.945 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:46:04:982.990 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:46:04:983.026 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: ClientInit 2026-03-23 Mon 05:46:04:983.049 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-1.:contrail-query-engine:0:Database count 7 2026-03-23 Mon 05:46:04:983.131 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:04:983.150 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:46:04:983.173 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: SANDESH: Send FAILED: 1774244764983020 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:04:983.181 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:46:04:983.190 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:04:983.197 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:46:04:983.207 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:46:04:983.213 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:46:04:983.347 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: SANDESH: Send FAILED: 1774244764983119 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 497257 calls = 59 average_bytes = 8428 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 167, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 58, _iter59->first = sent _iter59->second = 63, ] ] ] 2026-03-23 Mon 05:46:04:983.364 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:46:05:847.893 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:05:848.026 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:05:848.067 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:05:848.114 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:05:848.142 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:05:848.414 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:05:848.439 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:05:848.453 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:05:848.482 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:05:848.495 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:05:896.318 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:05:896.450 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:05:897.921 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:05:897.996 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:05:898.064 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:05:898.107 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:05:898.120 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:05:898.147 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:05:898.180 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:05:898.194 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:05:960.863 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:05:961.005 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:05:961.050 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:05:961.084 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:05:961.113 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:05:961.314 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:05:961.344 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:05:961.459 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:05:961.483 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:05:961.497 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:07:642.950 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.38:57516::10.0.0.50:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2026-03-23 Mon 05:46:07:642.988 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.38:57516::10.0.0.50:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:07:643.011 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:57516::10.0.0.50:8086(-1) OnSessionEvent TCP Connection Closed 2026-03-23 Mon 05:46:07:643.020 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:57516::10.0.0.50:8086(-1):EvTcpClose 2026-03-23 Mon 05:46:07:643.096 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2026-03-23 Mon 05:46:07:643.127 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: ClientInit : EvTcpClose 2026-03-23 Mon 05:46:07:643.273 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Return to idle with 1000 2026-03-23 Mon 05:46:07:643.283 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Idle 2026-03-23 Mon 05:46:07:643.346 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:07:643.507 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: SANDESH: Send FAILED: 1774244767643224 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:07:643.530 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:07:643.542 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:07:643.565 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: SANDESH: Send FAILED: 1774244767643255 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:07:643.572 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:07:643.581 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:07:643.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:07:643.751 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: SANDESH: Send FAILED: 1774244767643336 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:07:643.767 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:08:643.837 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:08:644.030 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-23 Mon 05:46:08:644.149 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:08:644.181 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244768643902 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:08:644.191 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:08:644.200 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:08:644.387 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244768644136 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:08:644.411 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:08:649.025 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:46:08:649.047 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:46:08:649.830 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:46:08:653.861 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:46:08:655.172 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:08:655.206 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Idle 2026-03-23 Mon 05:46:08:655.269 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:08:655.420 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244768655118 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 169, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 65, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:08:655.439 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:08:655.449 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:08:655.471 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244768655148 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:08:655.478 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:08:655.487 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:08:655.507 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:08:655.644 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: SANDESH: Send FAILED: 1774244768655259 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 170, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 65, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:08:655.662 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897880962816, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:10:848.471 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:10:848.567 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:10:848.603 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:10:848.631 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:10:848.657 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:10:848.849 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:10:848.892 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:10:848.908 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:10:848.922 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:10:849.808 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:10:896.509 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:10:896.639 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:10:898.048 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:10:898.154 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:10:898.176 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:10:898.224 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:10:898.237 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:10:898.266 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:10:898.299 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:10:898.324 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:10:961.395 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:10:961.563 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:10:961.625 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:10:961.664 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:10:961.706 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:10:961.953 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:10:962.003 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:10:962.025 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:10:962.067 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:10:962.088 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:15:848.966 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:15:849.102 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:15:849.144 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:15:849.171 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:15:849.420 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:15:849.484 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:15:849.508 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:15:849.548 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:15:849.850 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:15:850.205 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:15:896.719 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:15:896.856 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:15:898.187 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:15:898.239 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:15:898.258 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:15:898.290 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:15:898.356 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:15:898.461 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:15:898.507 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:15:898.529 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:15:962.030 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:15:962.150 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:15:962.185 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:15:962.212 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:15:962.239 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:15:962.393 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:15:962.428 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:15:962.441 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:15:962.453 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:15:962.467 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:20:849.483 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:20:849.612 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:20:849.646 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:20:849.708 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:20:849.862 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:20:849.891 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:20:849.906 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:20:849.955 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:20:850.241 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:20:850.493 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:20:896.936 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:20:897.087 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:20:898.290 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:20:898.413 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:20:898.442 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:20:898.496 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:20:898.530 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:20:898.598 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:20:898.617 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:20:898.668 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:20:962.457 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:20:962.577 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:20:962.611 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:20:962.635 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:20:962.662 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:20:962.922 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:20:962.966 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:20:962.980 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:20:962.993 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:20:963.016 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:21:411.743 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-03-23 Mon 05:46:21:411.810 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:21:411.912 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: SANDESH: Send FAILED: 1774244781411603 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 132 dequeues = 132 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 168 dequeues = 168 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244768655205 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244768655505 ] max_sm_queue_count = 6 ] 2026-03-23 Mon 05:46:21:411.935 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:21:411.947 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:21:412.123 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: SANDESH: Send FAILED: 1774244781411726 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 170, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:21:412.152 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:24:655.353 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:24:655.616 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:46:24:655.679 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:46:24:655.712 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:46:24:655.742 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:24:655.775 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: SANDESH: Send FAILED: 1774244784655441 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:24:655.784 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:24:655.793 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:46:24:655.804 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:46:24:655.902 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:24:655.946 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Idle 2026-03-23 Mon 05:46:24:656.011 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:24:656.203 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: SANDESH: Send FAILED: 1774244784655728 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 170, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 70, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:24:656.231 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:24:656.242 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:24:656.377 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: SANDESH: Send FAILED: 1774244784655857 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 170, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:24:656.393 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:24:656.403 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:24:656.424 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: SANDESH: Send FAILED: 1774244784655880 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:24:656.431 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:24:656.439 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:24:656.458 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:24:656.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: SANDESH: Send FAILED: 1774244784655999 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 171, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:24:656.606 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:25:849.973 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:25:850.118 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:25:850.182 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:25:850.241 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:25:850.443 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:25:850.476 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:25:850.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:25:850.520 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:25:850.561 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:25:850.769 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:25:897.155 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:25:897.309 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:25:898.452 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:25:898.514 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:25:898.536 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:25:898.571 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:25:898.626 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:25:898.674 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:25:898.697 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:25:898.729 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:25:963.008 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:25:963.145 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:25:963.195 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:25:963.230 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:25:963.258 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:25:963.431 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:25:963.483 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:25:963.512 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:25:963.527 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:25:963.540 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:30:850.509 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:30:850.641 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:30:850.676 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:30:850.703 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:30:850.801 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:30:851.016 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:30:851.046 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:30:851.059 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:30:851.073 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:30:851.085 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:30:897.365 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:30:897.485 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:30:898.547 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:30:898.884 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:30:898.903 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:30:898.947 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:30:898.979 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:30:899.012 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:30:899.027 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:30:899.040 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:30:963.516 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:30:963.636 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:30:963.678 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:30:963.712 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:30:963.741 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:30:964.021 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:30:964.070 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:30:964.096 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:30:964.111 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:30:964.126 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:35:851.089 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:35:851.192 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:35:851.230 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:35:851.261 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:35:851.297 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:35:851.488 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:35:851.521 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:35:851.569 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:35:851.610 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:35:851.649 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:35:897.557 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:35:897.679 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:35:898.919 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:35:898.981 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:35:899.039 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:35:899.082 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:35:899.094 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:35:899.122 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:35:899.155 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:35:899.170 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:35:964.116 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:35:964.225 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:35:964.261 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:35:964.293 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:35:964.335 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:35:964.684 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:35:964.778 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:35:964.824 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:35:964.843 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:35:964.857 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:40:851.556 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:40:851.674 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:40:851.728 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:40:851.769 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:40:851.799 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:40:851.978 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:40:852.006 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:40:852.020 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:40:852.041 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:40:852.119 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:40:897.739 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:40:897.870 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:40:899.040 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:40:899.177 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:40:899.207 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:40:899.264 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:40:899.281 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:40:899.319 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:40:899.364 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:40:899.383 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:40:964.762 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:40:964.898 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:40:964.942 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:40:964.975 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:40:965.006 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:40:965.235 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:40:965.262 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:40:965.275 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:40:965.423 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:40:965.447 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:45:852.060 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:45:852.185 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:45:852.218 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:45:852.251 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:45:852.280 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:45:852.458 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:45:852.481 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:45:852.503 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:45:852.548 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:45:852.587 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:45:897.937 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:45:898.058 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:45:899.224 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:45:899.310 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:45:899.329 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:45:899.363 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:45:899.398 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:45:899.433 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:45:899.445 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:45:899.475 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:45:965.310 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:45:965.449 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:45:965.485 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:45:965.519 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:45:965.546 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:45:965.692 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:45:965.718 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:45:965.757 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:45:965.782 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:45:965.796 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:50:852.528 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:50:890.381 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:50:890.439 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:50:890.466 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:50:890.493 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:50:890.611 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:50:890.689 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:50:890.711 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:50:890.731 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:50:890.751 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:50:898.121 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:50:898.249 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:50:899.348 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:50:899.488 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:50:899.520 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:50:899.563 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:50:899.611 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:50:899.631 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:50:899.643 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:50:899.685 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:50:965.764 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:50:965.953 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:50:966.001 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:50:966.040 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:50:966.078 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:50:966.181 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:50:966.218 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:50:966.238 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:50:966.256 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:50:966.376 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:51:412.711 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:51:412.803 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SANDESH: Send FAILED: 1774244811412582 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 139 dequeues = 139 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 178 dequeues = 178 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244784655944 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244784656457 ] max_sm_queue_count = 6 ] 2026-03-23 Mon 05:46:51:412.816 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:51:412.825 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:51:412.983 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: SANDESH: Send FAILED: 1774244811412677 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 171, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 75, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:51:413.006 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898263369472, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:54:656.690 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:46:54:656.892 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:46:54:657.001 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:54:657.029 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: SANDESH: Send FAILED: 1774244814656771 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:54:657.038 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:54:657.047 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:46:54:657.206 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: SANDESH: Send FAILED: 1774244814656990 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 171, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 77, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:54:657.229 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:54:657.920 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:46:54:657.940 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:46:54:658.565 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:46:54:658.574 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:46:54:658.697 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:46:54:658.724 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Idle 2026-03-23 Mon 05:46:54:658.783 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:54:658.938 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SANDESH: Send FAILED: 1774244814658638 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 171, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 79, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:54:658.960 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:54:658.970 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:54:658.992 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SANDESH: Send FAILED: 1774244814658661 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:46:54:658.999 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:46:54:659.008 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:46:54:659.016 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:54:659.162 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SANDESH: Send FAILED: 1774244814658773 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 151 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 172, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 79, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:46:54:659.180 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:46:55:890.712 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:46:55:890.819 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:46:55:890.853 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:46:55:890.880 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:46:55:890.908 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:46:55:891.048 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:46:55:891.073 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:46:55:891.123 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:46:55:891.138 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:46:55:891.160 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:46:55:898.313 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:46:55:898.421 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:46:55:899.526 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:46:55:899.595 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:46:55:899.645 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:46:55:899.709 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:46:55:899.727 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:46:55:899.767 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:46:55:899.779 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:46:55:899.812 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:46:55:966.245 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:46:55:966.348 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:46:55:966.382 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:46:55:966.416 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:46:55:966.448 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:46:55:966.664 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:46:55:966.689 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:46:55:966.709 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:46:55:966.723 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:46:55:966.735 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:46:56:406.575 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:46:56:406.651 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: SANDESH: Send FAILED: 1774244816406531 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2026-03-23 Mon 05:46:56:406.670 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2026-03-23 Mon 05:47:00:893.898 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:47:00:894.018 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:47:00:894.052 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:47:00:894.079 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:47:00:894.107 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:47:00:894.555 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:47:00:894.593 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:47:00:894.608 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:47:00:894.621 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:47:00:894.633 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:47:00:900.312 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:47:00:900.410 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:47:00:900.444 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:47:00:900.473 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:47:00:900.501 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:47:00:900.539 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:47:00:900.561 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:47:00:900.574 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:47:00:900.586 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:47:00:900.598 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:47:00:970.167 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:47:00:970.285 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:47:00:970.318 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:47:00:970.349 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:47:00:970.393 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:47:00:974.037 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:47:00:974.090 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:47:00:974.106 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:47:00:974.125 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:47:00:974.140 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:47:05:894.650 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:47:05:894.771 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:47:05:894.814 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:47:05:894.853 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:47:05:894.893 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:47:05:895.019 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:47:05:895.069 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:47:05:895.087 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:47:05:895.101 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:47:05:895.130 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:47:05:900.601 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:47:05:900.706 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:47:05:900.754 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:47:05:900.776 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:47:05:900.787 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:47:05:900.815 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:47:05:900.842 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:47:05:900.884 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:47:05:900.899 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:47:05:900.911 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:47:05:974.120 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:47:05:974.235 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:47:05:974.268 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:47:05:974.296 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:47:05:974.323 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:47:05:974.415 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:47:05:974.442 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:47:05:974.486 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:47:05:974.499 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:47:05:974.530 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:47:10:895.098 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:47:10:895.211 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:47:10:895.246 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:47:10:895.274 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:47:10:895.301 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:47:10:895.411 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ee420 2026-03-23 Mon 05:47:10:895.471 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ed1a0 2026-03-23 Mon 05:47:10:895.487 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ef6a0 2026-03-23 Mon 05:47:10:895.499 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eed60 2026-03-23 Mon 05:47:10:895.512 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27edae0 2026-03-23 Mon 05:47:10:900.797 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:47:10:900.906 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:47:10:900.970 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec870 2026-03-23 Mon 05:47:10:900.992 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ec340 2026-03-23 Mon 05:47:10:901.002 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:47:10:901.042 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:47:10:901.069 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:47:10:901.102 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27e51d0 2026-03-23 Mon 05:47:10:901.116 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27ebca0 2026-03-23 Mon 05:47:10:901.128 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27eb5f0 2026-03-23 Mon 05:47:10:979.580 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:47:10:979.704 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:47:10:979.736 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:47:10:979.769 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:47:10:979.799 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:47:10:980.030 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27effe0 2026-03-23 Mon 05:47:10:980.098 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f0920 2026-03-23 Mon 05:47:10:980.116 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f24e0 2026-03-23 Mon 05:47:10:980.130 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1ba0 2026-03-23 Mon 05:47:10:980.144 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: -1 0x27f1260 2026-03-23 Mon 05:47:15:895.559 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ee420 2026-03-23 Mon 05:47:15:895.665 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ed1a0 2026-03-23 Mon 05:47:15:895.700 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ef6a0 2026-03-23 Mon 05:47:15:895.728 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eed60 2026-03-23 Mon 05:47:15:895.753 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27edae0 2026-03-23 Mon 05:47:15:895.970 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ee420 2026-03-23 Mon 05:47:15:895.995 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:896.038 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:896.055 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ef6a0 2026-03-23 Mon 05:47:15:896.060 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:896.069 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:896.098 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ed1a0 2026-03-23 Mon 05:47:15:896.112 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:896.121 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:896.147 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27eed60 2026-03-23 Mon 05:47:15:896.152 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:896.160 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:896.168 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27edae0 2026-03-23 Mon 05:47:15:896.173 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:896.181 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:896.825 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:896.846 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:896.862 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:896.911 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:896.938 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:896.947 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:896.960 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:896.974 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:896.986 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:896.995 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:897.027 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:899.147 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:15:899.183 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: SANDESH: Send FAILED: 1774244835897056 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.38:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:15:899.192 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:15:899.346 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:899.374 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:899.390 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:899.405 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:901.011 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec870 2026-03-23 Mon 05:47:15:901.114 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ec340 2026-03-23 Mon 05:47:15:901.167 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ec870 2026-03-23 Mon 05:47:15:901.174 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:901.189 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:901.198 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ec340 2026-03-23 Mon 05:47:15:901.203 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:901.211 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:901.217 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27e51d0 2026-03-23 Mon 05:47:15:901.253 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27ebca0 2026-03-23 Mon 05:47:15:901.296 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27eb5f0 2026-03-23 Mon 05:47:15:901.360 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27e51d0 2026-03-23 Mon 05:47:15:901.367 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:901.377 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:901.385 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27ebca0 2026-03-23 Mon 05:47:15:901.390 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:901.398 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:901.405 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27eb5f0 2026-03-23 Mon 05:47:15:901.410 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:901.417 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:901.641 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:901.656 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:901.670 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:901.679 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:901.691 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:901.700 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:901.712 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:901.721 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:901.733 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:901.747 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:901.768 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:904.218 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:15:904.255 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: SANDESH: Send FAILED: 1774244835901794 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:15:904.264 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898267567872, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:15:904.307 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:904.330 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:904.345 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:904.359 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:980.130 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27effe0 2026-03-23 Mon 05:47:15:980.238 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f0920 2026-03-23 Mon 05:47:15:980.272 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f24e0 2026-03-23 Mon 05:47:15:980.301 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1ba0 2026-03-23 Mon 05:47:15:980.330 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x27f1260 2026-03-23 Mon 05:47:15:980.504 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27effe0 2026-03-23 Mon 05:47:15:980.516 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:980.546 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:980.649 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f0920 2026-03-23 Mon 05:47:15:980.813 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:980.831 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:980.843 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f24e0 2026-03-23 Mon 05:47:15:980.849 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:980.857 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:980.865 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f1ba0 2026-03-23 Mon 05:47:15:980.870 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:980.877 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:980.886 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: RAC_Connect status: 0 0x27f1260 2026-03-23 Mon 05:47:15:980.890 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:47:15:980.898 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2026-03-23 Mon 05:47:15:980.960 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:980.971 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:981.024 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:981.875 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:981.897 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:981.912 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:981.933 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:981.970 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:981.979 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:981.999 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2026-03-23 Mon 05:47:15:982.008 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2026-03-23 Mon 05:47:15:984.483 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:15:984.521 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: SANDESH: Send FAILED: 1774244835982039 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:15:984.531 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:15:989.949 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:989.991 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:990.007 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:15:990.021 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2026-03-23 Mon 05:47:21:412.913 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2026-03-23 Mon 05:47:21:412.990 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:21:413.086 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: SANDESH: Send FAILED: 1774244841412802 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 150 dequeues = 150 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 192 dequeues = 192 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244814658723 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244835984478 ] max_sm_queue_count = 6 ] 2026-03-23 Mon 05:47:21:413.113 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:21:413.131 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:21:413.278 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: SANDESH: Send FAILED: 1774244841412897 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 211 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 46 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 232, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 86, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:47:21:413.293 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898229782272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:24:658.836 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:47:24:659.075 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: Connect : Start Connect timer 10.0.0.33:8086 2026-03-23 Mon 05:47:24:659.178 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:24:659.207 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: SANDESH: Send FAILED: 1774244844658910 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.33:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:24:659.225 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:24:659.234 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:24:659.382 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: SANDESH: Send FAILED: 1774244844659163 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.33:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 211 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 232, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 88, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:47:24:659.397 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898242377472, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:24:659.538 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2026-03-23 Mon 05:47:24:659.576 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: -(-1):EvTcpConnectFail 2026-03-23 Mon 05:47:24:659.658 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2026-03-23 Mon 05:47:24:659.686 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Connect : EvTcpConnectFail 2026-03-23 Mon 05:47:24:659.804 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2026-03-23 Mon 05:47:24:659.837 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Idle 2026-03-23 Mon 05:47:24:659.897 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:24:660.072 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: SANDESH: Send FAILED: 1774244844659759 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 211 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 232, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 90, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:47:24:660.113 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:24:660.125 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:24:660.147 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: SANDESH: Send FAILED: 1774244844659783 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:24:660.157 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:24:660.166 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:47:24:660.187 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:24:660.382 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: SANDESH: Send FAILED: 1774244844659888 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 211 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 233, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 90, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:47:24:660.405 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898254972672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:51:413.236 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:51:413.362 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: SANDESH: Send FAILED: 1774244871413076 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 157 dequeues = 157 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 202 dequeues = 202 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244844659836 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244844660185 ] max_sm_queue_count = 6 ] 2026-03-23 Mon 05:47:51:413.377 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:51:413.387 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:47:51:413.534 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: SANDESH: Send FAILED: 1774244871413191 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 211 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 233, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 93, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:47:51:413.552 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898246575872, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:54:659.988 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2026-03-23 Mon 05:47:54:660.251 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Connect : Start Connect timer 10.0.0.38:8086 2026-03-23 Mon 05:47:54:660.350 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:54:660.379 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: SANDESH: Send FAILED: 1774244874660063 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:54:660.388 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:47:54:660.397 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:47:54:660.547 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: SANDESH: Send FAILED: 1774244874660337 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 211 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 53 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 233, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 95, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:47:54:660.582 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898250774272, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:47:54:662.527 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:57998::10.0.0.38:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2026-03-23 Mon 05:47:54:662.551 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:57998::10.0.0.38:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:47:54:662.574 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898411239552, Pid 1]: 10.0.0.38:57998::10.0.0.38:8086(-1):EvTcpConnected 2026-03-23 Mon 05:47:54:662.644 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:47:54:662.678 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:47:54:662.715 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: ClientInit 2026-03-23 Mon 05:47:54:662.733 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-1.:contrail-query-engine:0:Database count 8 2026-03-23 Mon 05:47:54:662.816 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:47:54:662.826 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:47:54:662.848 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: SANDESH: Send FAILED: 1774244874662708 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:47:54:662.856 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:47:54:662.864 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:47:54:662.871 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:47:54:662.882 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:47:54:662.897 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:47:54:663.047 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: SANDESH: Send FAILED: 1774244874662805 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 8 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 211 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 234, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 97, _iter59->first = sent _iter59->second = 64, ] ] ] 2026-03-23 Mon 05:47:54:663.063 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898259171072, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:47:54:664.470 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139898233980672, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:47:54:664.525 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:47:54:664.559 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:47:54:664.604 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:47:54:664.986 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 8 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 498830 calls = 60 average_bytes = 8313 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 157 dequeues = 157 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 202 dequeues = 202 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1774244844659836 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1774244844660185 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 19 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 153475 calls = 16 average_bytes = 9592 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 211 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 36 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 234, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 97, _iter59->first = sent _iter59->second = 64, ] ] max_sm_queue_count = 6 proxy seq 90 2026-03-23 Mon 05:47:54:665.030 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:47:54:665.064 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.38:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 52 2026-03-23 Mon 05:47:54:665.084 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:47:54:665.133 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 10 2026-03-23 Mon 05:47:54:665.152 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:47:54:665.163 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:47:54:665.201 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Established 2026-03-23 Mon 05:48:21:413.576 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897859979008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2026-03-23 Mon 05:49:21:414.182 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2026-03-23 Mon 05:50:21:417.147 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897855780608, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2026-03-23 Mon 05:50:22:760.652 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvStop in state Established 2026-03-23 Mon 05:50:22:760.784 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Idle 2026-03-23 Mon 05:50:22:760.923 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2026-03-23 Mon 05:50:22:760.952 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Processing scm::EvSandeshSend in state Idle 2026-03-23 Mon 05:50:22:761.121 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: SANDESH: Send FAILED: 1774245022760906 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1774244230986598 collector_name = collector_ip = 10.0.0.38:8086 collector_list= [ [ (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 674632 calls = 77 average_bytes = 8761 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 15 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 211 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 48 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 234, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 99, _iter59->first = sent _iter59->second = 84, ] ] ] 2026-03-23 Mon 05:50:22:761.149 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 139897872570112, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:50:47:613.600 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: http-server-port 8091 2026-03-23 Mon 05:50:47:613.662 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Max-tasks 16 2026-03-23 Mon 05:50:47:613.668 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Max-slice 100 2026-03-23 Mon 05:50:47:613.674 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Collectors 10.0.0.38:8086 2026-03-23 Mon 05:50:47:613.678 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Collectors 10.0.0.50:8086 2026-03-23 Mon 05:50:47:613.682 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Collectors 10.0.0.33:8086 2026-03-23 Mon 05:50:47:613.703 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: ROLE : Generator 2026-03-23 Mon 05:50:47:613.709 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: MODULE : contrail-query-engine 2026-03-23 Mon 05:50:47:613.713 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-5515-1. 2026-03-23 Mon 05:50:47:613.718 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: NODE TYPE : Database 2026-03-23 Mon 05:50:47:613.723 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: INSTANCE ID : 0 2026-03-23 Mon 05:50:47:613.730 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2026-03-23 Mon 05:50:47:613.737 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2026-03-23 Mon 05:50:47:616.573 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: HTTP Introspect Init 2026-03-23 Mon 05:50:47:616.675 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 138 2026-03-23 Mon 05:50:47:616.759 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Sandesh Http Server Port: 8091 2026-03-23 Mon 05:50:47:616.814 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2026-03-23 Mon 05:50:47:616.821 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2026-03-23 Mon 05:50:47:617.159 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Idle 2026-03-23 Mon 05:50:47:617.300 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: No Client: 1774245047617248 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245047617174 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:50:47:617.414 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2026-03-23 Mon 05:50:47:617.426 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2026-03-23 Mon 05:50:47:617.437 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Cassandra Servers: 10.0.0.38:9042 10.0.0.50:9042 10.0.0.33:9042 2026-03-23 Mon 05:50:47:620.330 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Processing scm::EvStart in state Idle 2026-03-23 Mon 05:50:47:620.419 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Disconnect 2026-03-23 Mon 05:50:47:620.506 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2026-03-23 Mon 05:50:47:620.984 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Connect : Start Connect timer 10.0.0.50:8086 2026-03-23 Mon 05:50:47:621.179 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:50:47:621.206 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: SANDESH: Send FAILED: 1774245047620410 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:50:47:621.216 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:50:47:621.228 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:50:47:621.413 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: SANDESH: Send FAILED: 1774245047620488 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245047617174 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:50:47:621.433 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:50:47:621.443 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:50:47:621.491 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: SANDESH: Send FAILED: 1774245047620541 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245047617174 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:50:47:621.499 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:50:47:621.507 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:50:47:621.519 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: SANDESH: Send FAILED: 1774245047620559 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:50:47:621.529 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:50:47:621.539 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:50:47:621.581 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: SANDESH: Send FAILED: 1774245047621158 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1774245047617174 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:50:47:621.589 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2026-03-23 Mon 05:50:47:688.212 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2026-03-23 Mon 05:50:47:690.165 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29ea580 2026-03-23 Mon 05:50:47:690.193 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.207 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.218 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29ef540 2026-03-23 Mon 05:50:47:690.223 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.230 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.239 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29efeb0 2026-03-23 Mon 05:50:47:690.243 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.250 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.258 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f0b70 2026-03-23 Mon 05:50:47:690.262 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.268 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.277 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f1830 2026-03-23 Mon 05:50:47:690.282 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.288 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.38 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.295 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f30b0 2026-03-23 Mon 05:50:47:690.300 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.306 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.314 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f3cf0 2026-03-23 Mon 05:50:47:690.318 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.324 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.332 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f24f0 2026-03-23 Mon 05:50:47:690.336 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.342 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.353 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f4930 2026-03-23 Mon 05:50:47:690.358 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.364 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.372 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f61b0 2026-03-23 Mon 05:50:47:690.376 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.383 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.391 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f5570 2026-03-23 Mon 05:50:47:690.395 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.401 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.50 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.409 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f6df0 2026-03-23 Mon 05:50:47:690.413 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.420 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.427 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f7a30 2026-03-23 Mon 05:50:47:690.432 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.438 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.445 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f8670 2026-03-23 Mon 05:50:47:690.450 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.456 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:690.464 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: RAC_Connect status: 0 0x29f92b0 2026-03-23 Mon 05:50:47:690.469 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: Connected to REDIS... 2026-03-23 Mon 05:50:47:690.477 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.33 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1028 2026-03-23 Mon 05:50:47:691.039 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376474863168, Pid 1]: Processing scm::EvSandeshSend in state Connect 2026-03-23 Mon 05:50:47:691.085 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376474863168, Pid 1]: SANDESH: Send FAILED: 1774245047690069 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:50:47:691.099 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376474863168, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2026-03-23 Mon 05:50:47:697.006 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:41912::10.0.0.50:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2026-03-23 Mon 05:50:47:697.033 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: 10.0.0.38:41912::10.0.0.50:8086(-1) OnSessionEvent TCP Connected 2026-03-23 Mon 05:50:47:697.041 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: 10.0.0.38:41912::10.0.0.50:8086(-1):EvTcpConnected 2026-03-23 Mon 05:50:47:697.381 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Processing scm::EvTcpConnected in state Connect 2026-03-23 Mon 05:50:47:697.410 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2026-03-23 Mon 05:50:47:697.450 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: ClientInit 2026-03-23 Mon 05:50:47:697.466 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-5515-1.:contrail-query-engine:0:Database count 1 2026-03-23 Mon 05:50:47:697.544 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:50:47:697.553 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:50:47:697.571 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SANDESH: Send FAILED: 1774245047697444 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2026-03-23 Mon 05:50:47:697.577 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:50:47:697.585 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:50:47:697.590 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2026-03-23 Mon 05:50:47:697.606 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:50:47:697.612 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2026-03-23 Mon 05:50:47:697.690 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SANDESH: Send FAILED: 1774245047697531 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774245047617174 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2026-03-23 Mon 05:50:47:697.711 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2026-03-23 Mon 05:50:47:699.220 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.241 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.254 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.261 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.272 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.279 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.300 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.313 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.324 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.336 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.38:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.396 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:699.418 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:699.430 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:699.560 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:699.583 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:699.626 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:50:47:699.656 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:50:47:699.689 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: SANDESH: Send FAILED: 1774245047699370 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2026-03-23 Mon 05:50:47:699.700 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:50:47:699.710 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:50:47:699.716 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:50:47:699.732 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: SANDESH: Send FAILED: 1774245047699414 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.50:6379::Down.10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:50:47:699.740 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:50:47:699.644 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.774 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.790 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.798 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.815 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.824 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.835 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.843 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.855 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.863 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.874 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.882 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.50:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.894 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.902 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.913 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.921 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.932 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.955 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.966 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1080 2026-03-23 Mon 05:50:47:699.974 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.33:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1087 2026-03-23 Mon 05:50:47:699.991 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:700.024 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:700.038 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:700.050 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:700.061 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:700.076 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:700.127 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:700.137 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:700.146 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:700.155 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 705 2026-03-23 Mon 05:50:47:700.187 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:50:47:700.205 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:50:47:700.223 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SANDESH: Send FAILED: 1774245047700015 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = 10.0.0.33:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:50:47:700.230 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:50:47:700.237 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2026-03-23 Mon 05:50:47:700.243 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2026-03-23 Mon 05:50:47:700.255 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SANDESH: Send FAILED: 1774245047700072 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2026-03-23 Mon 05:50:47:700.265 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2026-03-23 Mon 05:50:47:700.306 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: OnMessage control in state: ClientInit 2026-03-23 Mon 05:50:47:700.325 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2026-03-23 Mon 05:50:47:700.331 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: ClientInit : EvSandeshMessageRecv 2026-03-23 Mon 05:50:47:700.345 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Received Ctrl Message with size 0 2026-03-23 Mon 05:50:47:700.419 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1774245047617174 collector_name = collector_ip = 10.0.0.50:8086 collector_list= [ [ (*_iter40) = 10.0.0.50:8086, (*_iter40) = 10.0.0.38:8086, (*_iter40) = 10.0.0.33:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2026-03-23 Mon 05:50:47:700.449 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:50:47:700.466 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-5515-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.50:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.38:9042, (*_iter6) = 10.0.0.50:9042, (*_iter6) = 10.0.0.33:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.38:6379, (*_iter6) = 10.0.0.50:6379, (*_iter6) = 10.0.0.33:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2026-03-23 Mon 05:50:47:700.475 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2026-03-23 Mon 05:50:47:700.482 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:50:47:700.488 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2026-03-23 Mon 05:50:47:700.511 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: Established 2026-03-23 Mon 05:51:47:617.751 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375809295936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2026-03-23 Mon 05:52:47:618.319 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2026-03-23 Mon 05:53:47:618.885 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376491652672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2026-03-23 Mon 05:54:47:619.569 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375809295936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2026-03-23 Mon 05:55:47:620.056 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2026-03-23 Mon 05:56:47:620.671 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375800899136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2026-03-23 Mon 05:57:47:621.174 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375800899136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2026-03-23 Mon 05:58:47:621.677 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376491652672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2026-03-23 Mon 05:59:47:622.721 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375800899136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2026-03-23 Mon 06:00:47:623.979 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2026-03-23 Mon 06:01:47:624.535 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375805097536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2026-03-23 Mon 06:02:47:625.077 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375805097536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2026-03-23 Mon 06:03:47:625.941 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375805097536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2026-03-23 Mon 06:04:47:626.626 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375813494336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2026-03-23 Mon 06:05:47:627.138 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375821891136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2026-03-23 Mon 06:06:47:627.637 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376470664768, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2026-03-23 Mon 06:07:46:815.473 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:47146 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:07:47:628.266 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375805097536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2026-03-23 Mon 06:07:54:666.587 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:34820 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:08:47:628.768 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375821891136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2026-03-23 Mon 06:09:28:727.149 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:46116 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:09:44:675.907 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.38 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-03-23 Mon 06:09:44:676.205 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID e351fb86-267e-11f1-8397-00000a000026 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-03-23 Mon 06:09:44:676.266 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 :0:json query is: { end_time: 1774246184623285 , enqueue_time: 1774246184675388 , query_metadata: {"enqueue_time": 1774246184675295} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:44:676.425 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: Chunks: 2 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-03-23 Mon 06:09:44:676.441 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-03-23 Mon 06:09:44:676.489 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for e351fb86-267e-11f1-8397-00000a000026 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-03-23 Mon 06:09:44:676.557 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375805097536, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 :0:json query is: { end_time: 1774246184623285 , enqueue_time: 1774246184675388 , query_metadata: {"enqueue_time": 1774246184675295} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:44:773.701 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:44:773.752 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:44:773.761 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:44:773.767 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:44:773.773 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:44:773.780 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:44:773.786 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 UNION between tables of sizes 0 and 4 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:44:773.814 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 Resulting size of set 4 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:44:773.822 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 UNION between tables of sizes 4 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:44:773.831 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 Resulting size of set 4 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:44:774.073 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:774.187 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:774.286 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:774.385 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:783.964 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:44:784.008 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:44:784.016 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:44:784.022 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:44:784.028 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:44:784.035 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:44:784.041 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:44:784.048 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:44:784.054 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 UNION between tables of sizes 0 and 39 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:44:784.201 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QELog: e351fb86-267e-11f1-8397-00000a000026 Resulting size of set 39 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:44:784.287 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 :0:json query is: { end_time: 1774246184623285 , enqueue_time: 1774246184675388 , query_metadata: {"enqueue_time": 1774246184675295} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:44:784.467 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:784.592 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:784.706 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:784.813 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:784.955 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:785.056 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:785.155 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:785.251 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:785.348 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:785.442 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:785.533 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:785.626 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:785.723 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:785.816 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:785.925 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:786.033 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:786.125 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:786.218 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:786.309 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:786.397 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:786.487 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:786.579 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:786.668 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:786.762 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:786.854 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:786.974 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:787.070 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:787.165 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:787.256 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:787.345 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:787.438 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:787.531 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:787.623 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:787.724 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:787.814 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:787.916 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:788.031 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:788.295 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:788.388 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QEQueryLog: e351fb86-267e-11f1-8397-00000a000026 ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333 src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:44:788.708 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 43 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-03-23 Mon 06:09:44:789.050 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 43 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-03-23 Mon 06:09:44:789.189 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":6, "count":43} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-03-23 Mon 06:09:44:789.619 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375813494336, Pid 1]: [SYS_INFO]: QELog: Finished: QID e351fb86-267e-11f1-8397-00000a000026 Table ObjectVNTable Time(ms) 113 RedisTime(ms) 0 MergeTime(ms) 0 Rows 43 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-03-23 Mon 06:09:44:789.643 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376470664768, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5515-1. table = ObjectVNTable query_stats= [ qid = e351fb86-267e-11f1-8397-00000a000026 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-14817139:ctest-vn22-62662333", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] } post = time_span = 8 error = None chunks = 2 chunk_where_time = 97,0, 107,0, chunk_select_time = 0,0, 0,4, chunk_postproc_time = 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 113 rows = 43 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-03-23 Mon 06:09:44:789.706 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375813494336, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-03-23 Mon 06:09:47:629.293 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375834486336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2026-03-23 Mon 06:09:47:792.251 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.38 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 812 2026-03-23 Mon 06:09:47:792.591 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID e52d8628-267e-11f1-9309-00000a000026 src/contrail-analytics/contrail-query-engine/query.cc 1165 2026-03-23 Mon 06:09:47:792.628 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QEQueryLog: e52d8628-267e-11f1-9309-00000a000026 :0:json query is: { end_time: 1774246187781296 , enqueue_time: 1774246187791880 , query_metadata: {"enqueue_time": 1774246187791770} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "a63dda54-26d6-4035-ab8b-b822522507fe", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:47:792.766 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: Chunks: 2 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 936 2026-03-23 Mon 06:09:47:792.780 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 986 2026-03-23 Mon 06:09:47:792.820 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for e52d8628-267e-11f1-9309-00000a000026 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 994 2026-03-23 Mon 06:09:47:792.923 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: [SYS_INFO]: QEQueryLog: e52d8628-267e-11f1-9309-00000a000026 :0:json query is: { end_time: 1774246187781296 , enqueue_time: 1774246187791880 , query_metadata: {"enqueue_time": 1774246187791770} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "a63dda54-26d6-4035-ab8b-b822522507fe", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:47:832.219 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:47:832.263 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:47:832.272 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:47:832.279 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:47:832.293 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 UNION between tables of sizes 0 and 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:47:832.305 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:47:832.313 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:47:832.322 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:47:832.330 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:47:832.338 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:47:832.561 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QEQueryLog: e52d8628-267e-11f1-9309-00000a000026 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id a63dda54-26d6-4035-ab8b-b822522507fe src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:47:832.672 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: [SYS_INFO]: QEQueryLog: e52d8628-267e-11f1-9309-00000a000026 ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id a63dda54-26d6-4035-ab8b-b822522507fe src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:47:832.242 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:47:832.790 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:47:832.798 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:47:832.816 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:47:832.823 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:47:832.830 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:47:832.837 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:47:832.844 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:47:832.851 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 UNION between tables of sizes 0 and 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 59 2026-03-23 Mon 06:09:47:832.874 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: e52d8628-267e-11f1-9309-00000a000026 Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 66 2026-03-23 Mon 06:09:47:832.926 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QEQueryLog: e52d8628-267e-11f1-9309-00000a000026 :0:json query is: { end_time: 1774246187781296 , enqueue_time: 1774246187791880 , query_metadata: {"enqueue_time": 1774246187791770} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1774246172902724 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "a63dda54-26d6-4035-ab8b-b822522507fe", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2026-03-23 Mon 06:09:47:833.108 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QEQueryLog: e52d8628-267e-11f1-9309-00000a000026 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id a63dda54-26d6-4035-ab8b-b822522507fe src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:47:833.230 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QEQueryLog: e52d8628-267e-11f1-9309-00000a000026 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id a63dda54-26d6-4035-ab8b-b822522507fe src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:47:833.335 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QEQueryLog: e52d8628-267e-11f1-9309-00000a000026 ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id a63dda54-26d6-4035-ab8b-b822522507fe src/contrail-analytics/contrail-query-engine/select.cc 1022 2026-03-23 Mon 06:09:47:833.522 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 542 2026-03-23 Mon 06:09:47:833.590 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 551 2026-03-23 Mon 06:09:47:833.638 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375838684736, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 586 2026-03-23 Mon 06:09:47:833.927 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375817692736, Pid 1]: [SYS_INFO]: QELog: Finished: QID e52d8628-267e-11f1-9309-00000a000026 Table ObjectVMTable Time(ms) 41 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 661 2026-03-23 Mon 06:09:47:833.968 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-5515-1. table = ObjectVMTable query_stats= [ qid = e52d8628-267e-11f1-9309-00000a000026 where = [[{"name": "ObjectId", "value": "a63dda54-26d6-4035-ab8b-b822522507fe", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 2 chunk_where_time = 39,0, 39,0, chunk_select_time = 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 41 rows = 5 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 657 2026-03-23 Mon 06:09:47:834.120 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375800899136, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 699 2026-03-23 Mon 06:10:47:629.870 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375830287936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2026-03-23 Mon 06:11:47:630.570 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375821891136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2026-03-23 Mon 06:12:47:631.088 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2026-03-23 Mon 06:13:47:631.595 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375800899136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2026-03-23 Mon 06:14:47:632.162 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2026-03-23 Mon 06:15:47:632.666 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375809295936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2026-03-23 Mon 06:16:47:633.195 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376495851072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2026-03-23 Mon 06:17:47:633.790 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375813494336, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2026-03-23 Mon 06:18:47:634.423 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375826089536, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2026-03-23 Mon 06:19:47:635.049 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376500049472, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2026-03-23 Mon 06:20:47:635.544 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376491652672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2026-03-23 Mon 06:21:07:852.630 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:58736 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2026-03-23 Mon 06:21:47:636.992 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376487454272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2026-03-23 Mon 06:22:47:637.564 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140375809295936, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2026-03-23 Mon 06:23:20:510.490 UTC cn-jenkins-deploy-platform-ansible-os-5515-1 [Thread 140376507189376, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.38:8091::10.0.0.38:43866 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256