2025-01-27 Mon 00:50:50:248.034 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: http-server-port 8091 2025-01-27 Mon 00:50:50:248.130 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Max-tasks 16 2025-01-27 Mon 00:50:50:248.140 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Max-slice 100 2025-01-27 Mon 00:50:50:248.146 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Collectors 10.0.0.37:8086 2025-01-27 Mon 00:50:50:248.151 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Collectors 10.0.0.32:8086 2025-01-27 Mon 00:50:50:248.156 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Collectors 10.0.0.22:8086 2025-01-27 Mon 00:50:50:248.175 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: ROLE : Generator 2025-01-27 Mon 00:50:50:248.181 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-01-27 Mon 00:50:50:248.187 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2660-3. 2025-01-27 Mon 00:50:50:248.192 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: NODE TYPE : Database 2025-01-27 Mon 00:50:50:248.198 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: INSTANCE ID : 0 2025-01-27 Mon 00:50:50:248.203 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-01-27 Mon 00:50:50:248.209 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-01-27 Mon 00:50:50:248.865 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: HTTP Introspect Init 2025-01-27 Mon 00:50:50:248.984 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-01-27 Mon 00:50:50:249.045 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Sandesh Http Server Port: 8091 2025-01-27 Mon 00:50:50:249.088 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-01-27 Mon 00:50:50:249.097 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-01-27 Mon 00:50:50:249.171 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Idle 2025-01-27 Mon 00:50:50:249.328 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: No Client: 1737939050249254 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050249182 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:50:50:249.470 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-01-27 Mon 00:50:50:249.485 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-01-27 Mon 00:50:50:249.497 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Cassandra Servers: 10.0.0.37:9042 10.0.0.32:9042 10.0.0.22:9042 2025-01-27 Mon 00:50:50:254.879 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Processing scm::EvStart in state Idle 2025-01-27 Mon 00:50:50:254.952 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Disconnect 2025-01-27 Mon 00:50:50:255.029 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-01-27 Mon 00:50:50:255.228 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 00:50:50:255.313 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:255.335 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: SANDESH: Send FAILED: 1737939050254942 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-01-27 Mon 00:50:50:255.344 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:50:50:255.361 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:255.441 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: SANDESH: Send FAILED: 1737939050255018 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050249182 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:50:50:255.459 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:50:50:255.469 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:255.536 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: SANDESH: Send FAILED: 1737939050255076 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050249182 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:50:50:255.547 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:50:50:255.556 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:255.571 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: SANDESH: Send FAILED: 1737939050255095 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-01-27 Mon 00:50:50:255.578 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:50:50:255.585 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:255.646 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: SANDESH: Send FAILED: 1737939050255289 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050249182 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:50:50:255.657 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704102000384, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:50:50:265.921 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-01-27 Mon 00:50:50:265.961 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:50:50:266.043 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:50:50:266.079 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704088741632, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:50:50:266.110 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704088741632, Pid 1]: SANDESH: Send FAILED: 1737939050265997 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-01-27 Mon 00:50:50:266.120 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704088741632, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:50:55:269.230 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-01-27 Mon 00:50:55:269.281 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:50:55:269.319 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:00:271.181 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-01-27 Mon 00:51:00:271.240 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:00:271.263 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:05:273.133 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-01-27 Mon 00:51:05:273.187 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:05:273.211 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:10:355.966 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:10:356.029 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:10:356.052 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:15:384.140 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:15:384.194 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:15:384.217 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:20:416.989 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:20:417.042 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:20:417.065 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:25:464.966 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:25:465.036 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:25:465.062 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:30:490.078 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:30:490.131 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:30:490.155 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:35:519.016 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:35:519.062 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:35:519.085 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:40:569.261 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-01-27 Mon 00:51:40:569.310 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:40:569.343 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:45:609.468 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-01-27 Mon 00:51:45:609.520 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-01-27 Mon 00:51:50:626.946 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-01-27 Mon 00:51:55:637.800 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20a5830 2025-01-27 Mon 00:51:55:637.850 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:637.867 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:637.880 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20a1da0 2025-01-27 Mon 00:51:55:637.898 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:637.911 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:637.921 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20a6400 2025-01-27 Mon 00:51:55:637.927 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:637.935 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:637.944 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20a6b50 2025-01-27 Mon 00:51:55:637.949 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:637.957 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:637.966 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20a7470 2025-01-27 Mon 00:51:55:637.971 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:637.979 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:637.988 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20aabb0 2025-01-27 Mon 00:51:55:637.994 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:638.002 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:638.010 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20ab4f0 2025-01-27 Mon 00:51:55:638.016 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:638.024 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:638.033 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20a86b0 2025-01-27 Mon 00:51:55:638.038 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:638.046 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:638.055 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20a7d70 2025-01-27 Mon 00:51:55:638.060 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:638.068 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:638.081 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20abe30 2025-01-27 Mon 00:51:55:638.086 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:638.094 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:638.103 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20aa270 2025-01-27 Mon 00:51:55:638.108 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:638.116 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:638.125 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20a8ff0 2025-01-27 Mon 00:51:55:638.130 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:638.138 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:638.146 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20a9930 2025-01-27 Mon 00:51:55:638.152 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:638.159 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:638.168 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20ac770 2025-01-27 Mon 00:51:55:638.173 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:638.181 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:638.189 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: RAC_Connect status: 0 0x20ad0b0 2025-01-27 Mon 00:51:55:638.195 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: Connected to REDIS... 2025-01-27 Mon 00:51:55:638.203 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 00:51:55:638.230 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-01-27 Mon 00:51:55:638.238 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: -(-1):EvTcpConnectFail 2025-01-27 Mon 00:51:55:638.515 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:51:55:638.548 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SANDESH: Send FAILED: 1737939115637659 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-01-27 Mon 00:51:55:638.561 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:638.571 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-01-27 Mon 00:51:55:638.578 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Connect : EvTcpConnectFail 2025-01-27 Mon 00:51:55:638.702 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 00:51:55:638.752 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Idle 2025-01-27 Mon 00:51:55:638.814 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:638.920 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SANDESH: Send FAILED: 1737939115638656 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050249182 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 52, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:51:55:638.935 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:51:55:638.945 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:638.964 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SANDESH: Send FAILED: 1737939115638682 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-01-27 Mon 00:51:55:638.971 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:638.979 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 00:51:55:639.021 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:639.815 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SANDESH: Send FAILED: 1737939115638805 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050249182 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 53, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:51:55:639.850 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:51:55:639.946 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:639.993 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SANDESH: Send FAILED: 1737939115639878 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737939115638751 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737939115639020 ] max_sm_queue_count = 5 ] 2025-01-27 Mon 00:51:55:640.000 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:51:55:640.009 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:640.102 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SANDESH: Send FAILED: 1737939115639932 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050249182 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 53, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:51:55:640.115 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:51:55:641.675 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.688 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.704 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.714 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.728 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.737 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.750 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.759 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.775 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.784 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.798 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.807 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.819 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.828 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.841 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.850 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.862 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.871 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.883 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.892 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.906 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.915 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.927 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.937 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.951 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.960 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.972 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:641.981 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:641.994 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 00:51:55:642.003 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 00:51:55:643.313 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703995053824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:643.348 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703995053824, Pid 1]: SANDESH: Send FAILED: 1737939115642036 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 00:51:55:643.356 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703995053824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:645.303 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:645.770 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703995053824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:645.799 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703995053824, Pid 1]: SANDESH: Send FAILED: 1737939115645367 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 00:51:55:645.815 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703995053824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:645.926 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:645.951 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:645.975 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:645.991 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:646.020 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:646.786 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:646.815 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: SANDESH: Send FAILED: 1737939115646042 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 00:51:55:646.823 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:646.853 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:646.872 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:646.887 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:646.908 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:646.931 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:647.292 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703995053824, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 00:51:55:647.315 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703995053824, Pid 1]: SANDESH: Send FAILED: 1737939115646949 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 00:51:55:647.334 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703995053824, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:55:647.351 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:647.368 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:647.381 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:55:647.394 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 00:51:56:638.879 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703969871616, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 00:51:56:639.445 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703969871616, Pid 1]: Connect : Start Connect timer 10.0.0.32:8086 2025-01-27 Mon 00:51:56:639.562 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703969871616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:51:56:639.593 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703969871616, Pid 1]: SANDESH: Send FAILED: 1737939116638964 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 00:51:56:639.602 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703969871616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 00:51:56:639.612 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703969871616, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 00:51:56:639.724 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703969871616, Pid 1]: SANDESH: Send FAILED: 1737939116639548 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737939050249182 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:51:56:639.747 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703969871616, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 00:51:56:641.733 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:59454::10.0.0.32:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 00:51:56:641.763 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: 10.0.0.22:59454::10.0.0.32:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 00:51:56:641.772 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: 10.0.0.22:59454::10.0.0.32:8086(-1):EvTcpConnected 2025-01-27 Mon 00:51:56:641.885 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 00:51:56:641.901 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 00:51:56:641.932 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: ClientInit 2025-01-27 Mon 00:51:56:641.950 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-3.:contrail-query-engine:0:Database count 1 2025-01-27 Mon 00:51:56:642.034 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 00:51:56:642.043 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 00:51:56:642.065 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: SANDESH: Send FAILED: 1737939116641925 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 00:51:56:642.073 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 00:51:56:642.082 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 00:51:56:642.089 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 00:51:56:642.101 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 00:51:56:642.107 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 00:51:56:642.211 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: SANDESH: Send FAILED: 1737939116642022 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737939050249182 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 00:51:56:642.231 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 00:51:56:643.728 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: OnMessage control in state: ClientInit 2025-01-27 Mon 00:51:56:643.760 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-01-27 Mon 00:51:56:643.769 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-01-27 Mon 00:51:56:643.791 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: Received Ctrl Message with size 0 2025-01-27 Mon 00:51:56:643.932 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737939050249182 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737939115638751 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737939115639020 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2025-01-27 Mon 00:51:56:643.954 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 00:51:56:643.977 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-01-27 Mon 00:51:56:643.988 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 00:51:56:643.996 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-01-27 Mon 00:51:56:644.004 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 00:51:56:644.031 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: Established 2025-01-27 Mon 00:52:06:640.461 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: OnMessage control in state: Established 2025-01-27 Mon 00:52:06:640.579 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704011847424, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-01-27 Mon 00:52:06:640.620 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704011847424, Pid 1]: Established : EvSandeshMessageRecv 2025-01-27 Mon 00:52:06:640.671 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704011847424, Pid 1]: Received Ctrl Message with size 3 2025-01-27 Mon 00:52:06:640.690 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704011847424, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-01-27 Mon 00:52:06:640.701 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704011847424, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-01-27 Mon 00:52:06:640.714 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704011847424, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-01-27 Mon 00:52:06:640.722 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704011847424, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 00:52:25:640.236 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704088741632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-01-27 Mon 00:53:25:640.819 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704088741632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-01-27 Mon 00:54:25:641.394 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-01-27 Mon 00:54:32:577.040 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8091::10.0.0.22:39592 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-01-27 Mon 00:55:14:087.261 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704249870464, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8091::10.0.0.22:37376 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-01-27 Mon 00:55:25:642.817 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703969871616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-01-27 Mon 00:56:25:643.518 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703961474816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-01-27 Mon 00:57:25:644.118 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703273649920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-01-27 Mon 00:58:25:645.076 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703957276416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-01-27 Mon 00:59:25:645.610 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703957276416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-01-27 Mon 01:00:25:646.118 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703961474816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-01-27 Mon 01:01:25:646.698 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-01-27 Mon 01:02:25:647.278 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-01-27 Mon 01:03:25:647.861 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703965673216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-01-27 Mon 01:04:25:649.080 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-01-27 Mon 01:05:25:650.388 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703957276416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-01-27 Mon 01:06:25:650.946 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703957276416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-01-27 Mon 01:07:25:651.562 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703969871616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-01-27 Mon 01:08:25:652.136 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703961474816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-01-27 Mon 01:09:25:652.689 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-01-27 Mon 01:10:25:653.268 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703965673216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-01-27 Mon 01:11:25:653.732 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703965673216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-01-27 Mon 01:12:25:654.300 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-01-27 Mon 01:13:25:654.808 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703273649920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-01-27 Mon 01:14:25:655.308 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-01-27 Mon 01:15:25:655.975 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-01-27 Mon 01:16:25:656.452 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703961474816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-01-27 Mon 01:17:25:656.983 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703965673216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-01-27 Mon 01:18:25:657.635 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703273649920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-01-27 Mon 01:19:25:658.135 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704088741632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-01-27 Mon 01:20:25:658.859 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703969871616, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-01-27 Mon 01:21:25:659.423 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-01-27 Mon 01:22:25:659.990 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703273649920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-01-27 Mon 01:23:25:660.471 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-01-27 Mon 01:24:25:661.174 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703990855424, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-01-27 Mon 01:25:25:661.664 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703961474816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-01-27 Mon 01:26:25:663.492 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704088741632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-01-27 Mon 01:27:25:664.792 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703999252224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-01-27 Mon 01:28:25:665.247 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703957276416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-01-27 Mon 01:29:25:665.743 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-01-27 Mon 01:30:25:666.204 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-01-27 Mon 01:31:25:666.659 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703999252224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-01-27 Mon 01:32:25:667.192 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703999252224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-01-27 Mon 01:33:25:667.666 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703273649920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-01-27 Mon 01:34:25:668.112 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-01-27 Mon 01:35:25:668.674 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703961474816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-01-27 Mon 01:36:25:669.140 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-01-27 Mon 01:37:25:669.651 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-01-27 Mon 01:38:25:670.203 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-01-27 Mon 01:39:25:670.706 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703957276416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-01-27 Mon 01:40:25:671.306 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-01-27 Mon 01:41:25:671.832 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704084543232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-01-27 Mon 01:42:25:672.374 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-01-27 Mon 01:43:25:674.432 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703965673216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-01-27 Mon 01:44:25:674.972 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703999252224, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-01-27 Mon 01:45:25:675.840 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-01-27 Mon 01:46:25:676.318 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703961474816, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-01-27 Mon 01:47:25:676.855 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-01-27 Mon 01:48:25:677.356 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703957276416, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-01-27 Mon 01:49:25:677.829 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704084543232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-01-27 Mon 01:50:25:678.328 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-01-27 Mon 01:51:25:678.850 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704088741632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 60 without seqno , total = 0 2025-01-27 Mon 01:52:25:679.347 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 61 without seqno , total = 0 2025-01-27 Mon 01:53:25:679.819 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704088741632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 62 without seqno , total = 0 2025-01-27 Mon 01:54:25:680.302 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703965673216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 63 without seqno , total = 0 2025-01-27 Mon 01:55:25:681.152 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704007649024, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 64 without seqno , total = 0 2025-01-27 Mon 01:56:26:174.217 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703995053824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 65 without seqno , total = 0 2025-01-27 Mon 01:57:26:174.877 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703995053824, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 66 without seqno , total = 0 2025-01-27 Mon 01:58:26:175.423 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703273649920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 67 without seqno , total = 0 2025-01-27 Mon 01:59:26:175.974 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704084543232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 68 without seqno , total = 0 2025-01-27 Mon 02:00:26:176.486 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704088741632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 69 without seqno , total = 0 2025-01-27 Mon 02:01:26:177.004 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703273649920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 70 without seqno , total = 0 2025-01-27 Mon 02:02:26:177.624 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 71 without seqno , total = 0 2025-01-27 Mon 02:03:26:178.169 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703965673216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 72 without seqno , total = 0 2025-01-27 Mon 02:04:26:178.660 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139704003450624, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 73 without seqno , total = 0 2025-01-27 Mon 02:05:26:179.732 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703273649920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 74 without seqno , total = 0 2025-01-27 Mon 02:06:26:180.272 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703965673216, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 75 without seqno , total = 0 2025-01-27 Mon 02:06:36:870.652 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: Processing scm::EvStop in state Established 2025-01-27 Mon 02:06:36:870.753 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: Idle 2025-01-27 Mon 02:06:36:873.263 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:06:36:873.286 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:06:36:873.615 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: SANDESH: Send FAILED: 1737943596873235 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737939050249182 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3617607 calls = 378 average_bytes = 9570 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 75 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 300 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 101, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 379, ] ] ] 2025-01-27 Mon 02:06:36:873.651 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139703953078016, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:06:54:306.453 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: http-server-port 8091 2025-01-27 Mon 02:06:54:306.550 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Max-tasks 16 2025-01-27 Mon 02:06:54:306.558 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Max-slice 100 2025-01-27 Mon 02:06:54:306.565 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Collectors 10.0.0.37:8086 2025-01-27 Mon 02:06:54:306.570 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Collectors 10.0.0.32:8086 2025-01-27 Mon 02:06:54:306.575 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Collectors 10.0.0.22:8086 2025-01-27 Mon 02:06:54:306.594 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: ROLE : Generator 2025-01-27 Mon 02:06:54:306.600 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-01-27 Mon 02:06:54:306.606 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2660-3. 2025-01-27 Mon 02:06:54:306.611 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: NODE TYPE : Database 2025-01-27 Mon 02:06:54:306.616 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: INSTANCE ID : 0 2025-01-27 Mon 02:06:54:306.622 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-01-27 Mon 02:06:54:306.630 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-01-27 Mon 02:06:54:375.793 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: HTTP Introspect Init 2025-01-27 Mon 02:06:54:375.941 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-01-27 Mon 02:06:54:376.004 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Sandesh Http Server Port: 8091 2025-01-27 Mon 02:06:54:376.040 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-01-27 Mon 02:06:54:376.050 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-01-27 Mon 02:06:54:376.137 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Idle 2025-01-27 Mon 02:06:54:376.297 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: No Client: 1737943614376223 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:06:54:376.427 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-01-27 Mon 02:06:54:376.442 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-01-27 Mon 02:06:54:376.455 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Cassandra Servers: 10.0.0.37:9042 10.0.0.32:9042 10.0.0.22:9042 2025-01-27 Mon 02:06:54:391.315 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Processing scm::EvStart in state Idle 2025-01-27 Mon 02:06:54:391.395 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Disconnect 2025-01-27 Mon 02:06:54:391.474 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-01-27 Mon 02:06:54:391.686 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Connect : Start Connect timer 10.0.0.32:8086 2025-01-27 Mon 02:06:54:391.754 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:06:54:391.775 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: SANDESH: Send FAILED: 1737943614391386 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-01-27 Mon 02:06:54:391.785 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:06:54:391.793 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:06:54:391.872 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: SANDESH: Send FAILED: 1737943614391463 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:06:54:391.890 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:06:54:391.899 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:06:54:391.967 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: SANDESH: Send FAILED: 1737943614391534 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:06:54:391.980 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:06:54:391.989 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:06:54:392.003 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: SANDESH: Send FAILED: 1737943614391554 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-01-27 Mon 02:06:54:392.010 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:06:54:392.018 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:06:54:392.080 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: SANDESH: Send FAILED: 1737943614391743 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:06:54:392.091 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:06:54:475.883 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-01-27 Mon 02:06:54:480.064 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f58bb0 2025-01-27 Mon 02:06:54:480.097 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.129 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.143 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f594f0 2025-01-27 Mon 02:06:54:480.149 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.158 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.171 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f59e30 2025-01-27 Mon 02:06:54:480.177 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.185 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.207 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f53830 2025-01-27 Mon 02:06:54:480.214 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.222 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.240 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f5a770 2025-01-27 Mon 02:06:54:480.246 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.254 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.280 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f4fda0 2025-01-27 Mon 02:06:54:480.286 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.295 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.304 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f5b0b0 2025-01-27 Mon 02:06:54:480.310 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.318 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.326 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f54400 2025-01-27 Mon 02:06:54:480.331 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.339 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.360 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f54b50 2025-01-27 Mon 02:06:54:480.366 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.374 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.384 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f55d70 2025-01-27 Mon 02:06:54:480.389 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.397 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.405 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f55470 2025-01-27 Mon 02:06:54:480.410 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.430 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.440 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f56ff0 2025-01-27 Mon 02:06:54:480.446 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.454 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.465 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f566b0 2025-01-27 Mon 02:06:54:480.470 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.487 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.509 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f57930 2025-01-27 Mon 02:06:54:480.515 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.524 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:480.533 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f58270 2025-01-27 Mon 02:06:54:480.538 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:06:54:480.546 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:06:54:482.146 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:06:54:482.193 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: SANDESH: Send FAILED: 1737943614479940 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-01-27 Mon 02:06:54:482.206 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:06:54:491.770 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:55686::10.0.0.32:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:06:54:491.804 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:55686::10.0.0.32:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:06:54:491.812 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:55686::10.0.0.32:8086(-1):EvTcpConnected 2025-01-27 Mon 02:06:54:499.189 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:06:54:499.227 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:06:54:499.273 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: ClientInit 2025-01-27 Mon 02:06:54:499.292 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-3.:contrail-query-engine:0:Database count 1 2025-01-27 Mon 02:06:54:499.396 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:54:499.402 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:499.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:06:54:503.770 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: SANDESH: Send FAILED: 1737943614499265 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-01-27 Mon 02:06:54:503.781 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:06:54:503.796 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:54:503.805 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:06:54:503.818 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:54:503.825 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:06:54:503.938 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: SANDESH: Send FAILED: 1737943614499384 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 16 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 18, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:06:54:503.958 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:06:54:503.731 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.578 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:504.604 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.615 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.639 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.649 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.662 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.671 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.684 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.693 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.707 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.716 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.729 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.738 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.751 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.764 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.782 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.794 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.807 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.816 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.829 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.841 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.854 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.863 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.876 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.885 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.898 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.907 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.921 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:06:54:504.930 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:06:54:504.969 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:504.989 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.005 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.022 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.049 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.073 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.095 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.109 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.122 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.139 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.156 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.170 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.182 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.195 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:06:54:505.270 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:54:505.285 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:06:54:505.309 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SANDESH: Send FAILED: 1737943614504561 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:06:54:505.317 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:06:54:505.326 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:54:505.333 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:06:54:505.350 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SANDESH: Send FAILED: 1737943614504959 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down.10.0.0.32:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:06:54:505.356 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:06:54:505.368 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:54:505.375 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:06:54:505.391 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SANDESH: Send FAILED: 1737943614505067 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:06:54:505.397 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:06:54:505.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:06:54:505.412 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:06:54:505.428 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SANDESH: Send FAILED: 1737943614505090 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:06:54:505.434 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:06:55:500.161 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: 10.0.0.32:8086 EvConnectTimerExpired in state ClientInit 2025-01-27 Mon 02:06:55:500.228 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2025-01-27 Mon 02:06:55:500.239 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: ClientInit : EvConnectTimerExpired 2025-01-27 Mon 02:06:55:500.437 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Return to idle with 5000 2025-01-27 Mon 02:06:55:500.455 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Idle 2025-01-27 Mon 02:06:55:500.561 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:06:55:500.711 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943615500320 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 1, ] ] ] 2025-01-27 Mon 02:06:55:500.739 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:06:55:500.750 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:06:55:500.773 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943615500354 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:06:55:500.780 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:06:55:500.789 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:06:55:500.800 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:06:55:500.912 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943615500543 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 12, _iter59->first = sent _iter59->second = 1, ] ] ] 2025-01-27 Mon 02:06:55:500.928 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:06:59:480.416 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:06:59:480.541 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: SANDESH: Send FAILED: 1737943619480379 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. table_info= [ [ _iter16->first = MessageTablev2 [ reads = 1 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = ObjectValueTable [ reads = 1 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = SessionTable [ reads = 1 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = StatsTableByStrStrTagV3 [ reads = 0 read_fails = 1 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = StatsTableV4 [ reads = 1 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = SystemObjectTable [ reads = 2 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 1 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 2 available_connections = 2 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-01-27 Mon 02:06:59:480.557 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-01-27 Mon 02:07:00:500.542 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:07:00:500.744 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 02:07:00:500.836 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:07:00:500.863 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: SANDESH: Send FAILED: 1737943620500607 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:07:00:500.872 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:07:00:500.882 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:07:00:501.023 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: SANDESH: Send FAILED: 1737943620500825 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 1, ] ] ] 2025-01-27 Mon 02:07:00:501.047 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:07:00:503.516 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:57430::10.0.0.37:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:07:00:503.538 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:57430::10.0.0.37:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:07:00:503.546 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:57430::10.0.0.37:8086(-1):EvTcpConnected 2025-01-27 Mon 02:07:00:503.612 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:07:00:503.630 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:07:00:503.660 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: ClientInit 2025-01-27 Mon 02:07:00:503.677 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-3.:contrail-query-engine:0:Database count 2 2025-01-27 Mon 02:07:00:503.762 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:07:00:503.772 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:07:00:503.794 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SANDESH: Send FAILED: 1737943620503653 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:07:00:503.808 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:07:00:503.817 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:07:00:503.824 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:07:00:503.835 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:07:00:503.917 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:07:00:504.048 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SANDESH: Send FAILED: 1737943620503752 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 64, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 1, ] ] ] 2025-01-27 Mon 02:07:00:504.065 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:07:00:505.841 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: OnMessage control in state: ClientInit 2025-01-27 Mon 02:07:00:505.872 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-01-27 Mon 02:07:00:505.881 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-01-27 Mon 02:07:00:505.903 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Received Ctrl Message with size 0 2025-01-27 Mon 02:07:00:506.032 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 64, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 1, ] ] proxy seq 9 2025-01-27 Mon 02:07:00:506.057 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:07:00:506.080 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-01-27 Mon 02:07:00:506.091 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:07:00:506.149 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3. table_info= [ [ _iter16->first = MessageTablev2 [ reads = 1 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = ObjectValueTable [ reads = 1 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = SessionTable [ reads = 1 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = StatsTableByStrStrTagV3 [ reads = 0 read_fails = 1 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = StatsTableV4 [ reads = 1 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], _iter16->first = SystemObjectTable [ reads = 2 read_fails = 0 writes = 0 write_fails = 0 write_back_pressure_fails = 0 ], ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 1 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 2 available_connections = 2 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 1 2025-01-27 Mon 02:07:00:506.165 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:07:00:506.172 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 02:07:00:506.197 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Established 2025-01-27 Mon 02:07:54:378.634 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-01-27 Mon 02:08:54:383.364 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028660352768, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-01-27 Mon 02:09:21:869.415 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f55470 2025-01-27 Mon 02:09:21:869.506 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:869.552 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f54b50 2025-01-27 Mon 02:09:21:869.564 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:869.585 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f54400 2025-01-27 Mon 02:09:21:869.596 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:869.617 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f4fda0 2025-01-27 Mon 02:09:21:869.627 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:869.895 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-01-27 Mon 02:09:21:869.910 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f53830 2025-01-27 Mon 02:09:21:869.920 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:871.590 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:09:21:871.679 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:09:21:871.697 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:09:21:871.711 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:09:21:871.751 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:57430::10.0.0.37:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:09:21:871.767 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:57430::10.0.0.37:8086(-1):EvTcpClose 2025-01-27 Mon 02:09:21:872.390 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvTcpClose in state Established 2025-01-27 Mon 02:09:21:872.410 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Established : EvTcpClose 2025-01-27 Mon 02:09:21:872.712 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-01-27 Mon 02:09:21:873.996 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:21:874.165 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SANDESH: Send FAILED: 1737943761872516 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 2 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = cn-jenkins-deploy-platform-ansible-os-2660-1. collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 143024 calls = 17 average_bytes = 8413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 70, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 19, ] ] ] 2025-01-27 Mon 02:09:21:874.186 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:21:874.197 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:21:874.221 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SANDESH: Send FAILED: 1737943761872545 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:09:21:874.231 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:21:874.241 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-01-27 Mon 02:09:21:874.250 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:21:874.379 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SANDESH: Send FAILED: 1737943761873979 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 143024 calls = 17 average_bytes = 8413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 70, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 19, ] ] ] 2025-01-27 Mon 02:09:21:874.394 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:21:873.781 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:09:21:876.933 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:40006::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:09:21:876.960 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:40006::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:09:21:876.968 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:40006::10.0.0.22:8086(-1):EvTcpConnected 2025-01-27 Mon 02:09:21:879.367 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:09:21:879.393 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:09:21:879.432 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: ClientInit 2025-01-27 Mon 02:09:21:879.455 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-3.:contrail-query-engine:0:Database count 3 2025-01-27 Mon 02:09:21:879.681 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:21:879.700 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:09:21:879.843 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: SANDESH: Send FAILED: 1737943761879425 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:09:21:879.865 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:09:21:879.876 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:21:879.884 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:09:21:879.897 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:21:879.903 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:21:880.039 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: SANDESH: Send FAILED: 1737943761879664 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 143024 calls = 17 average_bytes = 8413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 71, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 19, ] ] ] 2025-01-27 Mon 02:09:21:880.059 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:21:952.549 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: OnMessage control in state: ClientInit 2025-01-27 Mon 02:09:21:952.616 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-01-27 Mon 02:09:21:952.629 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-01-27 Mon 02:09:21:952.657 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: Received Ctrl Message with size 0 2025-01-27 Mon 02:09:21:952.881 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 143024 calls = 17 average_bytes = 8413 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 35 dequeues = 35 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 43 dequeues = 43 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Established state_since = 1737943620506196 last_state = Established last_event = scm::EvSandeshSend last_event_at = 1737943704380750 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 14 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 116102 calls = 12 average_bytes = 9675 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 71, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 19, ] ] max_sm_queue_count = 6 proxy seq 21 2025-01-27 Mon 02:09:21:952.916 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:09:21:952.944 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 16 2025-01-27 Mon 02:09:21:952.957 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:09:21:952.989 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 3 2025-01-27 Mon 02:09:21:953.002 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:09:21:953.010 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 02:09:21:953.045 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: Established 2025-01-27 Mon 02:09:21:959.287 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f56ff0 2025-01-27 Mon 02:09:21:959.343 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:959.384 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f566b0 2025-01-27 Mon 02:09:21:959.396 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:959.418 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f57930 2025-01-27 Mon 02:09:21:959.429 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:959.449 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f58270 2025-01-27 Mon 02:09:21:959.459 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:959.480 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-01-27 Mon 02:09:21:959.495 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f55d70 2025-01-27 Mon 02:09:21:959.504 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:21:961.507 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:09:21:961.548 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:09:21:961.566 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:09:21:961.580 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:09:21:962.254 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:09:22:037.032 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f5b0b0 2025-01-27 Mon 02:09:22:037.097 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:22:037.147 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f5a770 2025-01-27 Mon 02:09:22:037.158 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:22:037.304 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f59e30 2025-01-27 Mon 02:09:22:037.317 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:22:037.343 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f594f0 2025-01-27 Mon 02:09:22:037.353 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:22:037.381 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-01-27 Mon 02:09:22:037.388 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Disconnect status: -1 0x1f58bb0 2025-01-27 Mon 02:09:22:037.396 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-01-27 Mon 02:09:22:039.963 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:09:22:040.000 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:09:22:040.035 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:09:22:040.049 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:09:22:040.061 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:09:22:105.771 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:40006::10.0.0.22:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:09:22:105.808 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:40006::10.0.0.22:8086(-1):EvTcpClose 2025-01-27 Mon 02:09:22:105.901 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvTcpClose in state Established 2025-01-27 Mon 02:09:22:105.939 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Established : EvTcpClose 2025-01-27 Mon 02:09:22:106.209 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Connect : Start Connect timer 10.0.0.32:8086 2025-01-27 Mon 02:09:22:106.305 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:22:106.454 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SANDESH: Send FAILED: 1737943762106040 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 3 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = cn-jenkins-deploy-platform-ansible-os-2660-3. collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 199226 calls = 23 average_bytes = 8662 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 28, ] ] ] 2025-01-27 Mon 02:09:22:106.486 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:22:106.506 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:22:106.529 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SANDESH: Send FAILED: 1737943762106075 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:22:106.536 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:22:106.545 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-01-27 Mon 02:09:22:106.572 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:22:106.699 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SANDESH: Send FAILED: 1737943762106292 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 199226 calls = 23 average_bytes = 8662 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 25, _iter59->first = sent _iter59->second = 28, ] ] ] 2025-01-27 Mon 02:09:22:106.712 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:22:109.335 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:48038::10.0.0.32:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:09:22:109.357 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:48038::10.0.0.32:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:09:22:109.365 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:48038::10.0.0.32:8086(-1):EvTcpConnected 2025-01-27 Mon 02:09:22:112.034 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:09:22:112.057 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:09:22:112.093 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: ClientInit 2025-01-27 Mon 02:09:22:112.111 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-3.:contrail-query-engine:0:Database count 4 2025-01-27 Mon 02:09:22:112.198 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:22:112.207 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:09:22:112.230 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: SANDESH: Send FAILED: 1737943762112085 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:22:112.242 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:09:22:112.251 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:22:112.258 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:09:22:112.269 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:22:112.275 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:22:112.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: SANDESH: Send FAILED: 1737943762112188 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 199226 calls = 23 average_bytes = 8662 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 28, ] ] ] 2025-01-27 Mon 02:09:22:112.423 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:22:123.359 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:48038::10.0.0.32:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:09:22:123.388 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:48038::10.0.0.32:8086(-1):EvTcpClose 2025-01-27 Mon 02:09:22:136.257 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-01-27 Mon 02:09:22:136.288 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: ClientInit : EvTcpClose 2025-01-27 Mon 02:09:22:136.413 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Return to idle with 5000 2025-01-27 Mon 02:09:22:136.423 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Idle 2025-01-27 Mon 02:09:22:136.481 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:22:136.642 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SANDESH: Send FAILED: 1737943762136368 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200799 calls = 24 average_bytes = 8366 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 29, ] ] ] 2025-01-27 Mon 02:09:22:136.666 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:22:136.677 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:22:136.699 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SANDESH: Send FAILED: 1737943762136400 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:22:136.706 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:22:136.715 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:09:22:136.740 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:22:136.866 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SANDESH: Send FAILED: 1737943762136472 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200799 calls = 24 average_bytes = 8366 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 29, ] ] ] 2025-01-27 Mon 02:09:22:138.556 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:24:385.091 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:24:385.185 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: SANDESH: Send FAILED: 1737943764384974 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 62 dequeues = 62 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 79 dequeues = 79 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737943762136422 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737943762136739 ] max_sm_queue_count = 6 ] 2025-01-27 Mon 02:09:24:385.199 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:24:385.209 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:24:385.357 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: SANDESH: Send FAILED: 1737943764385066 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200799 calls = 24 average_bytes = 8366 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 29, ] ] ] 2025-01-27 Mon 02:09:24:385.381 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:26:871.709 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:09:26:871.939 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:09:26:871.982 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:09:26:872.009 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:09:26:872.270 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:09:26:872.334 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:09:26:872.351 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:09:26:872.364 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:09:26:874.452 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:09:26:874.698 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:09:26:961.604 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:09:26:961.772 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:09:26:961.823 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:09:26:961.856 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:09:26:962.117 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:09:26:962.176 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:09:26:962.194 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:09:26:962.317 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:09:26:962.670 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:09:27:040.060 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:09:27:040.285 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:09:27:040.355 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:09:27:040.401 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:09:27:040.414 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:09:27:040.448 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:09:27:040.475 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:09:27:040.518 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:09:27:040.534 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:09:27:040.546 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:09:27:137.242 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:09:27:137.429 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 02:09:27:137.558 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:27:137.587 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: SANDESH: Send FAILED: 1737943767137297 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:27:137.596 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:27:137.605 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:27:137.748 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: SANDESH: Send FAILED: 1737943767137544 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200799 calls = 24 average_bytes = 8366 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 29, ] ] ] 2025-01-27 Mon 02:09:27:137.768 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:27:140.426 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:59096::10.0.0.37:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:09:27:140.452 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:59096::10.0.0.37:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:09:27:140.460 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:59096::10.0.0.37:8086(-1):EvTcpConnected 2025-01-27 Mon 02:09:27:140.553 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:09:27:140.607 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:09:27:140.665 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: ClientInit 2025-01-27 Mon 02:09:27:140.698 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-3.:contrail-query-engine:0:Database count 5 2025-01-27 Mon 02:09:27:140.827 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:27:140.842 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:09:27:140.878 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SANDESH: Send FAILED: 1737943767140653 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:27:140.906 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:09:27:140.920 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:27:140.930 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:09:27:140.947 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:27:140.957 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:27:141.164 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SANDESH: Send FAILED: 1737943767140812 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200799 calls = 24 average_bytes = 8366 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 29, ] ] ] 2025-01-27 Mon 02:09:27:141.186 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:27:226.854 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:09:29:140.795 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: 10.0.0.37:8086 EvConnectTimerExpired in state ClientInit 2025-01-27 Mon 02:09:29:140.878 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2025-01-27 Mon 02:09:29:140.917 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: ClientInit : EvConnectTimerExpired 2025-01-27 Mon 02:09:29:141.130 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Return to idle with 5000 2025-01-27 Mon 02:09:29:141.143 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Idle 2025-01-27 Mon 02:09:29:141.212 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:29:141.373 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: SANDESH: Send FAILED: 1737943769141016 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 202372 calls = 25 average_bytes = 8094 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 39, _iter59->first = sent _iter59->second = 30, ] ] ] 2025-01-27 Mon 02:09:29:141.399 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:29:141.413 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:29:141.437 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: SANDESH: Send FAILED: 1737943769141050 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:29:141.447 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:29:141.457 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:09:29:141.556 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:29:141.695 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: SANDESH: Send FAILED: 1737943769141199 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 202372 calls = 25 average_bytes = 8094 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 39, _iter59->first = sent _iter59->second = 30, ] ] ] 2025-01-27 Mon 02:09:29:141.711 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:31:872.393 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:09:31:872.604 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:09:31:872.686 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:09:31:872.725 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:09:31:872.815 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:09:31:872.857 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:09:31:872.877 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:09:31:872.892 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:09:31:874.749 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:09:31:874.939 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:09:31:962.231 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:09:31:962.420 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:09:31:962.480 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:09:31:962.809 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:09:31:962.854 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:09:31:962.870 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:09:31:962.883 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:09:31:963.264 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:09:32:040.441 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:09:32:040.686 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:09:32:040.729 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:09:32:040.776 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:09:32:040.816 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:09:32:040.854 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:09:32:040.903 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:09:32:040.927 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:09:32:040.967 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:09:32:040.985 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:09:32:226.966 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:09:32:227.372 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:09:34:141.686 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:09:34:141.929 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-01-27 Mon 02:09:34:142.038 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:34:142.126 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: SANDESH: Send FAILED: 1737943774141750 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:34:142.137 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:34:142.148 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:34:142.286 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: SANDESH: Send FAILED: 1737943774142015 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 202372 calls = 25 average_bytes = 8094 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 42, _iter59->first = sent _iter59->second = 30, ] ] ] 2025-01-27 Mon 02:09:34:142.308 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:34:144.290 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:58446::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:09:34:144.311 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:58446::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:09:34:144.319 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:58446::10.0.0.22:8086(-1):EvTcpConnected 2025-01-27 Mon 02:09:34:144.527 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:09:34:144.539 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:09:34:144.571 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: ClientInit 2025-01-27 Mon 02:09:34:144.588 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-3.:contrail-query-engine:0:Database count 6 2025-01-27 Mon 02:09:34:144.663 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:34:144.672 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:09:34:144.695 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: SANDESH: Send FAILED: 1737943774144564 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:34:144.703 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:09:34:144.712 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:34:144.719 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:09:34:144.736 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:09:34:144.743 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:09:34:144.880 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: SANDESH: Send FAILED: 1737943774144653 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 202372 calls = 25 average_bytes = 8094 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 86, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 44, _iter59->first = sent _iter59->second = 30, ] ] ] 2025-01-27 Mon 02:09:34:144.894 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028630963968, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:09:34:463.785 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.22:58446::10.0.0.22:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-01-27 Mon 02:09:34:463.958 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.22:58446::10.0.0.22:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:09:34:464.011 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:58446::10.0.0.22:8086(-1) OnSessionEvent TCP Connection Closed 2025-01-27 Mon 02:09:34:464.020 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:58446::10.0.0.22:8086(-1):EvTcpClose 2025-01-27 Mon 02:09:34:464.667 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-01-27 Mon 02:09:34:464.698 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: ClientInit : EvTcpClose 2025-01-27 Mon 02:09:34:464.998 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Return to idle with 5000 2025-01-27 Mon 02:09:34:465.027 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Idle 2025-01-27 Mon 02:09:34:465.140 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:34:465.368 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943774464793 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 46, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:09:34:465.396 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:34:465.411 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:34:465.444 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943774464862 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:34:465.461 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:34:465.474 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:09:34:465.522 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:34:465.675 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943774465123 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 46, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:09:34:465.690 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:36:872.912 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:09:36:873.061 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:09:36:873.115 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:09:36:873.165 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:09:36:873.235 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:09:36:873.272 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:09:36:873.286 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:09:36:873.313 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:09:36:874.973 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:09:36:875.184 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:09:36:962.924 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:09:36:963.101 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:09:36:963.155 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:09:36:963.299 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:09:36:963.510 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:09:36:963.579 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:09:36:963.596 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:09:36:963.613 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:09:37:040.780 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:09:37:041.075 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:09:37:041.110 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:09:37:041.148 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:09:37:041.180 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:09:37:041.210 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:09:37:041.243 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:09:37:041.259 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:09:37:041.272 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:09:37:041.285 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:09:37:227.438 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:09:37:227.774 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:09:39:465.740 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:09:39:466.039 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Connect : Start Connect timer 10.0.0.32:8086 2025-01-27 Mon 02:09:39:467.042 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-01-27 Mon 02:09:39:467.064 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: -(-1):EvTcpConnectFail 2025-01-27 Mon 02:09:39:467.135 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:39:467.178 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943779465835 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:39:467.188 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:39:467.199 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-01-27 Mon 02:09:39:467.207 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Connect : EvTcpConnectFail 2025-01-27 Mon 02:09:39:467.316 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:09:39:467.349 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Idle 2025-01-27 Mon 02:09:39:467.424 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:39:467.593 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943779467102 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 49, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:09:39:467.616 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:39:467.638 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:39:467.846 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943779467269 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:09:39:467.868 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:39:467.879 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:39:467.902 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943779467296 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:39:467.909 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:39:467.918 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:09:39:467.938 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:39:468.089 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943779467410 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:09:39:468.108 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:41:873.311 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:09:41:873.477 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:09:41:873.535 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:09:41:873.572 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:09:41:873.839 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:09:41:873.935 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:09:41:873.961 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:09:41:873.984 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:09:41:875.257 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:09:41:875.596 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:09:41:963.613 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:09:41:965.427 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:09:41:965.467 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:09:41:965.506 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:09:41:965.848 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:09:41:965.934 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:09:41:965.953 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:09:41:965.970 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:09:42:041.148 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:09:42:041.313 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:09:42:041.339 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:09:42:041.374 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:09:42:041.426 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:09:42:041.461 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:09:42:041.507 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:09:42:041.525 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:09:42:041.538 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:09:42:041.550 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:09:42:227.849 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:09:42:228.284 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:09:46:873.962 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:09:46:874.138 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:09:46:874.185 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:09:46:874.217 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:09:46:874.390 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:09:46:874.454 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:09:46:874.474 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:09:46:874.497 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:09:46:875.656 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:09:46:875.957 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:09:46:965.963 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:09:46:966.119 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:09:46:966.158 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:09:46:966.187 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:09:46:966.451 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:09:46:966.512 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:09:46:966.533 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:09:46:966.549 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:09:47:041.379 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:09:47:041.560 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:09:47:041.591 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:09:47:041.627 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:09:47:041.664 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:09:47:041.679 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:09:47:041.690 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:09:47:041.717 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:09:47:041.748 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:09:47:041.762 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:09:47:228.409 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:09:47:228.858 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:09:47:467.425 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:09:47:915.951 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 02:09:47:916.067 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:47:916.097 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943787915825 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:47:916.108 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:47:916.118 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:09:47:916.292 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943787916053 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 54, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:09:47:916.317 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:47:916.354 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-01-27 Mon 02:09:47:916.363 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: -(-1):EvTcpConnectFail 2025-01-27 Mon 02:09:47:916.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-01-27 Mon 02:09:47:916.415 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Connect : EvTcpConnectFail 2025-01-27 Mon 02:09:47:916.523 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:09:47:916.620 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Idle 2025-01-27 Mon 02:09:47:916.685 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:47:916.820 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943787916471 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:09:47:916.840 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:47:916.850 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:47:916.871 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943787916503 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:09:47:916.878 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:09:47:916.887 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:09:47:916.897 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:47:917.020 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: SANDESH: Send FAILED: 1737943787916676 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 56, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:09:47:917.035 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028639360768, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:51:874.503 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:09:51:874.660 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:09:51:874.803 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:09:51:874.833 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:09:51:874.963 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:09:51:875.023 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:09:51:875.043 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:09:51:875.057 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:09:51:876.012 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:09:51:876.199 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:09:51:966.550 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:09:51:966.735 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:09:51:966.771 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:09:51:966.798 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:09:51:967.137 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:09:51:967.192 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:09:51:967.208 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:09:51:967.221 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:09:52:041.639 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:09:52:041.810 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:09:52:041.838 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:09:52:041.873 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:09:52:041.902 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:09:52:041.929 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:09:52:041.960 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:09:52:041.975 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:09:52:041.988 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:09:52:042.005 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:09:52:228.946 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:09:52:229.319 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:09:54:385.377 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-01-27 Mon 02:09:54:385.463 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:54:385.591 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SANDESH: Send FAILED: 1737943794385226 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 90 dequeues = 90 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 121 dequeues = 121 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737943787916618 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737943787916896 ] max_sm_queue_count = 6 ] 2025-01-27 Mon 02:09:54:385.632 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:54:385.649 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:54:385.902 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: SANDESH: Send FAILED: 1737943794385357 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 59, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:09:54:385.922 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028643559168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:09:56:875.053 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:09:56:875.226 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:09:56:875.264 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:09:56:875.289 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:09:56:875.545 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:09:56:875.579 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:09:56:875.594 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:09:56:875.607 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:09:56:876.244 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:09:56:876.514 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:09:56:967.224 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:09:56:967.370 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:09:56:967.408 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:09:56:967.437 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:09:56:967.744 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:09:56:967.803 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:09:56:967.820 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:09:56:967.839 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:09:57:041.899 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:09:57:042.092 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:09:57:042.124 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:09:57:042.172 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:09:57:042.211 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:09:57:042.254 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:09:57:042.301 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:09:57:042.323 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:09:57:042.342 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:09:57:042.373 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:09:57:229.446 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:09:57:229.811 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:09:59:483.392 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028269283072, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:09:59:483.476 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028269283072, Pid 1]: SANDESH: Send FAILED: 1737943799483350 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-01-27 Mon 02:09:59:483.489 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028269283072, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-01-27 Mon 02:10:01:875.628 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:01:875.774 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:01:875.816 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:01:875.844 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:01:876.045 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:01:876.079 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:01:876.101 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:01:876.116 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:01:876.571 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:01:876.782 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:01:967.829 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:01:967.970 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:01:968.005 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:01:968.047 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:01:968.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:01:968.471 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:01:968.519 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:01:968.541 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:02:042.188 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:02:042.380 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:02:042.409 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:02:042.446 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:02:042.485 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:02:042.512 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:02:042.523 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:02:042.551 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:02:042.581 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:02:042.595 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:02:229.878 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:02:230.311 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:03:916.737 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:10:03:917.011 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-01-27 Mon 02:10:03:917.149 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:10:03:917.183 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SANDESH: Send FAILED: 1737943803916823 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:10:03:917.193 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:10:03:917.203 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:10:03:917.347 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SANDESH: Send FAILED: 1737943803917134 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 62, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:10:03:917.363 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:03:918.377 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-01-27 Mon 02:10:03:918.401 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: -(-1):EvTcpConnectFail 2025-01-27 Mon 02:10:03:918.443 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-01-27 Mon 02:10:03:918.457 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Connect : EvTcpConnectFail 2025-01-27 Mon 02:10:03:918.569 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:10:03:918.599 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Idle 2025-01-27 Mon 02:10:03:918.657 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:03:918.794 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943803918527 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 64, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:10:03:918.813 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:03:918.823 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:03:918.844 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943803918549 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:10:03:918.851 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:10:03:918.859 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:10:03:918.879 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:03:919.001 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943803918648 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 64, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:10:03:919.019 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:06:876.130 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:06:876.291 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:06:876.336 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:06:876.367 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:06:876.562 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:06:876.704 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:06:876.735 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:06:876.766 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:06:876.812 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:06:876.972 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:06:968.524 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:06:968.682 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:06:968.757 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:06:968.785 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:06:968.931 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:06:969.020 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:06:969.036 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:06:969.056 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:07:042.453 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:07:042.632 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:07:042.660 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:07:042.693 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:07:042.723 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:07:042.749 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:07:042.781 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:07:042.796 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:07:042.809 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:07:042.821 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:07:230.396 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:07:230.770 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:11:876.652 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:11:876.863 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:11:876.902 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:11:876.934 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:11:876.998 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:11:877.110 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:11:877.160 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:11:877.175 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:11:877.208 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:11:877.229 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:11:969.012 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:11:969.214 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:11:969.250 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:11:969.279 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:11:969.610 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:11:969.759 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:11:969.785 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:11:969.803 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:12:042.716 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:12:042.951 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:12:042.985 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:12:043.022 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:12:043.053 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:12:043.082 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:12:043.142 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:12:043.161 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:12:043.174 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:12:043.186 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:12:230.852 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:12:231.289 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:16:877.189 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:16:877.314 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:16:877.349 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:16:877.376 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:16:877.401 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:16:877.452 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:16:877.484 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:16:877.524 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:16:877.544 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:16:877.557 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:16:969.704 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:16:969.853 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:16:969.888 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:16:969.915 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:16:970.251 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:16:970.306 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:16:970.323 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:16:970.337 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:17:043.018 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:17:043.210 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:17:043.239 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:17:043.279 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:17:043.314 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:17:043.342 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:17:043.374 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:17:043.393 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:17:043.419 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:17:043.435 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:17:231.362 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:17:231.654 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:21:877.544 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:21:877.674 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:21:877.714 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:21:877.747 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:21:877.774 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:21:877.838 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:21:877.867 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:21:877.903 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:21:877.924 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:21:877.938 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:21:970.342 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:21:970.485 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:21:970.530 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:21:970.560 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:21:970.859 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:21:970.893 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:21:970.907 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:21:970.919 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:22:043.273 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:22:043.428 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:22:043.453 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:22:043.507 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:22:043.532 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:22:043.563 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:22:043.590 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:22:043.620 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:22:043.635 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:22:043.647 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:22:231.722 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:22:232.058 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:24:386.433 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:24:386.563 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: SANDESH: Send FAILED: 1737943824386265 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 98 dequeues = 98 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 132 dequeues = 132 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737943803918598 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737943803918877 ] max_sm_queue_count = 6 ] 2025-01-27 Mon 02:10:24:386.580 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:24:386.590 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:24:386.733 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: SANDESH: Send FAILED: 1737943824386389 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 67, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:10:24:386.759 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:26:877.919 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:26:878.058 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:26:878.100 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:26:878.133 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:26:878.161 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:26:878.237 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:26:878.274 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:26:878.296 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:26:878.317 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:26:878.330 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:26:970.943 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:26:971.153 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:26:971.201 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:26:971.251 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:26:971.522 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:26:971.565 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:26:971.581 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:26:971.594 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:27:043.518 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:27:043.720 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:27:043.758 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:27:043.796 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:27:043.828 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:27:043.858 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:27:043.892 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:27:043.908 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:27:043.921 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:27:043.934 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:27:232.144 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:27:232.533 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:31:878.328 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:31:878.487 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:31:878.543 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:31:878.572 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:31:878.598 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:31:878.745 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:31:878.777 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:31:878.803 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:31:878.817 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:31:878.838 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:31:971.596 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:31:971.744 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:31:971.780 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:31:971.828 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:31:972.129 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:31:972.201 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:31:972.258 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:31:972.273 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:32:043.790 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:32:043.957 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:32:043.987 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:32:044.035 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:32:044.067 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:32:044.095 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:32:044.128 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:32:044.145 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:32:044.159 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:32:044.171 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:32:232.601 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:32:232.974 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:33:918.703 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:10:33:918.895 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Connect : Start Connect timer 10.0.0.32:8086 2025-01-27 Mon 02:10:33:918.996 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:10:33:919.024 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: SANDESH: Send FAILED: 1737943833918765 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.32:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:10:33:919.042 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:10:33:919.051 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:10:33:919.230 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: SANDESH: Send FAILED: 1737943833918981 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.32:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 69, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:10:33:919.248 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:33:919.694 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-01-27 Mon 02:10:33:919.715 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: -(-1):EvTcpConnectFail 2025-01-27 Mon 02:10:33:919.771 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-01-27 Mon 02:10:33:919.788 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Connect : EvTcpConnectFail 2025-01-27 Mon 02:10:33:919.939 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:10:33:919.969 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Idle 2025-01-27 Mon 02:10:33:920.041 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:33:920.182 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: SANDESH: Send FAILED: 1737943833919863 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:10:33:920.199 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:33:920.209 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:33:920.231 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: SANDESH: Send FAILED: 1737943833919918 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:10:33:920.296 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:10:33:920.307 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:10:33:920.326 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:33:920.463 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: SANDESH: Send FAILED: 1737943833920030 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:10:33:920.511 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028260886272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:36:878.824 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:36:878.958 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:36:879.006 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:36:879.034 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:36:879.061 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:36:879.205 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:36:879.237 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:36:879.251 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:36:879.294 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:36:879.309 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:36:972.233 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:36:972.373 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:36:972.412 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:36:972.441 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:36:972.684 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:36:972.724 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:36:972.738 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:36:972.751 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:37:044.025 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:37:044.191 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:37:044.215 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:37:044.248 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:37:044.278 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:37:044.313 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:37:044.328 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:37:044.340 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:37:044.349 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:37:044.387 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:37:233.043 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:37:233.382 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:41:879.270 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:41:879.415 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:41:879.451 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:41:879.478 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:41:879.520 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:41:879.561 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:41:879.612 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:41:879.627 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:41:879.641 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:41:879.670 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:41:972.753 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:41:972.868 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:41:972.901 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:41:972.932 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:41:973.150 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:41:973.178 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:41:973.194 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:41:973.207 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:42:044.276 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:42:044.454 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:42:044.481 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:42:044.535 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:42:044.566 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:42:044.595 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:42:044.640 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:42:044.656 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:42:044.669 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:42:044.682 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:42:233.451 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:42:233.835 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:46:879.646 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:46:879.798 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:46:879.833 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:46:879.862 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:46:879.887 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:46:880.099 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:46:880.200 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:46:880.222 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:46:880.238 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:46:880.252 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:46:973.223 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:46:973.361 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:46:973.400 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:46:973.425 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:46:973.856 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:46:973.956 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:46:973.978 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:46:973.993 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:47:044.533 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:47:044.716 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:47:044.743 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:47:044.789 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:47:044.821 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:47:044.848 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:47:044.881 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:47:044.897 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:47:044.910 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:47:044.922 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:47:233.926 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:47:234.364 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:51:880.261 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:51:880.486 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:51:880.547 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:51:880.581 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:51:880.608 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:51:880.913 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:51:881.017 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:51:881.044 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:51:881.067 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:51:881.086 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:51:974.002 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:51:974.218 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:51:974.269 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:51:974.308 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:51:974.550 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:51:974.654 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:51:974.673 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:51:974.690 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:52:044.808 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:52:045.076 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:52:045.115 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:52:045.165 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:52:045.209 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:52:045.250 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:52:045.298 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:52:045.320 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:52:045.338 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:52:045.356 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:52:234.456 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:52:234.908 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:54:386.642 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-01-27 Mon 02:10:54:386.685 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:54:386.751 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: SANDESH: Send FAILED: 1737943854386528 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 105 dequeues = 105 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 142 dequeues = 142 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737943833919968 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737943833920324 ] max_sm_queue_count = 6 ] 2025-01-27 Mon 02:10:54:386.772 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:54:386.781 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:54:386.922 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: SANDESH: Send FAILED: 1737943854386626 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 92, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 74, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:10:54:386.936 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:10:56:882.377 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:10:56:882.513 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:10:56:882.547 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:10:56:882.573 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:10:56:882.600 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:10:56:882.751 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55470 2025-01-27 Mon 02:10:56:882.819 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54b50 2025-01-27 Mon 02:10:56:882.838 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f54400 2025-01-27 Mon 02:10:56:882.853 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f4fda0 2025-01-27 Mon 02:10:56:882.866 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f53830 2025-01-27 Mon 02:10:56:975.168 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:10:56:975.290 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:10:56:975.326 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:10:56:975.359 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:10:56:975.758 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f55d70 2025-01-27 Mon 02:10:56:975.795 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f57930 2025-01-27 Mon 02:10:56:975.810 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f566b0 2025-01-27 Mon 02:10:56:975.824 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58270 2025-01-27 Mon 02:10:57:045.181 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:10:57:045.323 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5b0b0 2025-01-27 Mon 02:10:57:045.353 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:10:57:045.392 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:10:57:045.421 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:10:57:045.458 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f5a770 2025-01-27 Mon 02:10:57:045.473 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f59e30 2025-01-27 Mon 02:10:57:045.486 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f594f0 2025-01-27 Mon 02:10:57:045.507 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:10:57:045.539 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f58bb0 2025-01-27 Mon 02:10:57:235.013 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:10:57:235.315 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: -1 0x1f56ff0 2025-01-27 Mon 02:10:59:487.080 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:10:59:487.186 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: SANDESH: Send FAILED: 1737943859486796 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-01-27 Mon 02:10:59:487.200 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-01-27 Mon 02:11:01:882.854 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55470 2025-01-27 Mon 02:11:01:883.022 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54b50 2025-01-27 Mon 02:11:01:883.066 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f54400 2025-01-27 Mon 02:11:01:883.094 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f4fda0 2025-01-27 Mon 02:11:01:883.142 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f53830 2025-01-27 Mon 02:11:01:883.225 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f55470 2025-01-27 Mon 02:11:01:883.233 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:01:883.278 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:01:883.290 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f54b50 2025-01-27 Mon 02:11:01:883.296 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:01:883.304 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:01:883.326 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f54400 2025-01-27 Mon 02:11:01:883.331 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:01:883.339 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:01:883.368 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f4fda0 2025-01-27 Mon 02:11:01:883.374 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:01:883.382 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:01:883.410 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f53830 2025-01-27 Mon 02:11:01:883.428 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:01:883.436 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:01:883.815 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:01:883.868 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:01:883.905 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:01:883.915 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:01:883.941 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:01:884.012 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:01:884.057 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:01:884.119 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: SANDESH: Send FAILED: 1737943861883971 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.32:6379::Down.10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:01:884.132 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:01:884.386 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:01:884.407 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:01:884.429 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:01:884.442 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:01:884.462 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:01:884.487 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:01:884.536 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:01:884.562 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:01:884.586 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:01:975.843 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f55d70 2025-01-27 Mon 02:11:01:975.979 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f57930 2025-01-27 Mon 02:11:01:976.024 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f566b0 2025-01-27 Mon 02:11:01:976.054 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58270 2025-01-27 Mon 02:11:01:976.365 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f55d70 2025-01-27 Mon 02:11:01:976.374 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:01:976.411 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:01:976.456 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f566b0 2025-01-27 Mon 02:11:01:976.462 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:01:976.472 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:01:976.480 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f58270 2025-01-27 Mon 02:11:01:976.485 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:01:976.493 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:01:976.521 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f57930 2025-01-27 Mon 02:11:01:976.528 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:01:976.540 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:01:976.960 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:01:976.994 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:01:977.009 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:01:977.019 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:01:977.043 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:01:977.762 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:01:977.785 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:01:977.795 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:01:977.812 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:01:977.821 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:01:977.794 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:01:977.838 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:01:977.898 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:01:977.907 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943861977072 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.22:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:01:977.918 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:02:046.039 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5b0b0 2025-01-27 Mon 02:11:02:046.194 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f5a770 2025-01-27 Mon 02:11:02:046.237 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f59e30 2025-01-27 Mon 02:11:02:046.271 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f594f0 2025-01-27 Mon 02:11:02:046.309 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f58bb0 2025-01-27 Mon 02:11:02:046.352 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f5b0b0 2025-01-27 Mon 02:11:02:046.359 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:046.380 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:046.391 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f5a770 2025-01-27 Mon 02:11:02:046.396 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:046.404 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:046.412 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f59e30 2025-01-27 Mon 02:11:02:046.418 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:046.425 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:046.433 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f594f0 2025-01-27 Mon 02:11:02:046.438 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:046.446 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:046.454 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f58bb0 2025-01-27 Mon 02:11:02:046.459 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:046.467 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:046.620 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:046.637 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:046.660 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:046.676 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:046.691 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:046.704 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:046.713 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:046.778 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:046.794 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:046.809 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:046.821 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:046.834 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:046.843 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:046.857 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:046.870 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:02:046.810 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:02:047.402 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: SANDESH: Send FAILED: 1737943862046741 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:02:047.413 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028651955968, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:02:235.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x1f56ff0 2025-01-27 Mon 02:11:02:235.934 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: RAC_Connect status: 0 0x1f56ff0 2025-01-27 Mon 02:11:02:235.951 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:11:02:235.975 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:11:02:236.345 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:11:02:236.361 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:11:02:236.383 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:11:03:920.098 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:11:03:920.376 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 02:11:03:920.518 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:11:03:920.551 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-01-27 Mon 02:11:03:920.590 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: -(-1):EvTcpConnectFail 2025-01-27 Mon 02:11:03:920.553 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943863920169 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:03:920.629 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:03:920.639 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:11:03:920.815 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943863920504 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 80, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:11:03:920.851 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:03:920.862 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-01-27 Mon 02:11:03:920.870 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Connect : EvTcpConnectFail 2025-01-27 Mon 02:11:03:920.979 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-01-27 Mon 02:11:03:921.036 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Idle 2025-01-27 Mon 02:11:03:921.094 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:03:921.278 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943863920935 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 82, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:11:03:921.297 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:03:921.307 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:03:921.340 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943863920955 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:03:921.347 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:03:921.356 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:11:03:921.385 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:03:921.536 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: SANDESH: Send FAILED: 1737943863921084 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 82, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:11:03:921.553 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028622567168, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:24:386.953 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:24:387.052 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SANDESH: Send FAILED: 1737943884386815 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 116 dequeues = 116 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 156 dequeues = 156 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737943863921035 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737943863921384 ] max_sm_queue_count = 6 ] 2025-01-27 Mon 02:11:24:387.066 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:24:387.076 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:11:24:387.225 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SANDESH: Send FAILED: 1737943884386928 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 85, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:11:24:387.249 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:33:921.735 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-01-27 Mon 02:11:33:921.952 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Connect : Start Connect timer 10.0.0.22:8086 2025-01-27 Mon 02:11:33:922.087 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:11:33:922.130 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: SANDESH: Send FAILED: 1737943893921813 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:33:922.142 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:11:33:922.154 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:11:33:922.378 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: SANDESH: Send FAILED: 1737943893922068 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 87, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:11:33:922.398 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028265084672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:11:33:929.649 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:44924::10.0.0.22:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-01-27 Mon 02:11:33:929.676 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:44924::10.0.0.22:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:11:33:929.684 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028804024448, Pid 1]: 10.0.0.22:44924::10.0.0.22:8086(-1):EvTcpConnected 2025-01-27 Mon 02:11:33:930.240 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:11:33:930.252 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:11:33:930.287 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: ClientInit 2025-01-27 Mon 02:11:33:930.300 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-3.:contrail-query-engine:0:Database count 7 2025-01-27 Mon 02:11:33:930.386 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:11:33:930.402 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:11:33:930.426 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: SANDESH: Send FAILED: 1737943893930279 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:11:33:930.434 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:11:33:930.444 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:11:33:930.451 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:11:33:930.462 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:11:33:930.469 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:11:33:930.613 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: SANDESH: Send FAILED: 1737943893930376 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 89, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-01-27 Mon 02:11:33:930.632 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028647757568, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:11:33:937.613 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: OnMessage control in state: ClientInit 2025-01-27 Mon 02:11:33:937.662 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-01-27 Mon 02:11:33:937.672 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-01-27 Mon 02:11:33:937.698 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Received Ctrl Message with size 0 2025-01-27 Mon 02:11:33:937.919 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 2034 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203945 calls = 26 average_bytes = 7844 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 116 dequeues = 116 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 156 dequeues = 156 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1737943863921035 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1737943863921384 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 14 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 116102 calls = 12 average_bytes = 9675 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 89, _iter59->first = sent _iter59->second = 31, ] ] max_sm_queue_count = 6 proxy seq 62 2025-01-27 Mon 02:11:33:937.951 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:11:33:937.985 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.22:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 44 2025-01-27 Mon 02:11:33:938.000 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:11:33:938.032 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 5 2025-01-27 Mon 02:11:33:938.042 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:11:33:938.050 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 02:11:33:938.081 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Established 2025-01-27 Mon 02:11:54:387.912 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-01-27 Mon 02:12:54:389.938 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-01-27 Mon 02:13:54:391.662 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028277679872, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-01-27 Mon 02:14:54:393.734 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028269283072, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-01-27 Mon 02:15:54:395.068 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-01-27 Mon 02:16:54:395.665 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028281878272, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-01-27 Mon 02:17:54:397.072 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028656154368, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-01-27 Mon 02:18:02:663.913 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvStop in state Established 2025-01-27 Mon 02:18:02:664.017 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Idle 2025-01-27 Mon 02:18:02:664.117 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-01-27 Mon 02:18:02:664.128 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-01-27 Mon 02:18:02:664.270 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: SANDESH: Send FAILED: 1737944282664105 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1737943614376147 collector_name = collector_ip = 10.0.0.22:8086 collector_list= [ [ (*_iter40) = 10.0.0.32:8086, (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, ] ] rx_socket_stats= [ bytes = 3051 calls = 6 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 577026 calls = 63 average_bytes = 9159 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 40 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 51 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 154, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 91, _iter59->first = sent _iter59->second = 71, ] ] ] 2025-01-27 Mon 02:18:02:664.293 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 140028626765568, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:18:30:823.505 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: http-server-port 8091 2025-01-27 Mon 02:18:30:823.629 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Max-tasks 16 2025-01-27 Mon 02:18:30:823.638 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Max-slice 100 2025-01-27 Mon 02:18:30:823.644 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Collectors 10.0.0.37:8086 2025-01-27 Mon 02:18:30:823.649 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Collectors 10.0.0.32:8086 2025-01-27 Mon 02:18:30:823.655 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Collectors 10.0.0.22:8086 2025-01-27 Mon 02:18:30:823.684 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: ROLE : Generator 2025-01-27 Mon 02:18:30:823.690 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-01-27 Mon 02:18:30:823.696 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2660-3. 2025-01-27 Mon 02:18:30:823.702 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: NODE TYPE : Database 2025-01-27 Mon 02:18:30:823.707 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: INSTANCE ID : 0 2025-01-27 Mon 02:18:30:823.713 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-01-27 Mon 02:18:30:823.718 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-01-27 Mon 02:18:30:825.288 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: HTTP Introspect Init 2025-01-27 Mon 02:18:30:825.415 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-01-27 Mon 02:18:30:825.481 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Sandesh Http Server Port: 8091 2025-01-27 Mon 02:18:30:825.520 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-01-27 Mon 02:18:30:825.540 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-01-27 Mon 02:18:30:825.626 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Idle 2025-01-27 Mon 02:18:30:825.792 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: No Client: 1737944310825715 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737944310825637 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:18:30:830.119 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-01-27 Mon 02:18:30:830.158 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-01-27 Mon 02:18:30:830.177 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Cassandra Servers: 10.0.0.37:9042 10.0.0.32:9042 10.0.0.22:9042 2025-01-27 Mon 02:18:30:835.272 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Processing scm::EvStart in state Idle 2025-01-27 Mon 02:18:30:847.643 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Disconnect 2025-01-27 Mon 02:18:30:847.752 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-01-27 Mon 02:18:30:848.022 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Connect : Start Connect timer 10.0.0.37:8086 2025-01-27 Mon 02:18:30:848.103 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:18:30:848.126 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: SANDESH: Send FAILED: 1737944310847612 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-01-27 Mon 02:18:30:848.135 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:18:30:848.144 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:18:30:848.230 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: SANDESH: Send FAILED: 1737944310847739 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737944310825637 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:18:30:848.248 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:18:30:848.258 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:18:30:848.324 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: SANDESH: Send FAILED: 1737944310847804 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737944310825637 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.32:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:18:30:848.336 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:18:30:848.345 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:18:30:848.360 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: SANDESH: Send FAILED: 1737944310847835 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-01-27 Mon 02:18:30:848.366 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:18:30:848.374 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:18:30:848.440 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: SANDESH: Send FAILED: 1737944310848092 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1737944310825637 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.32:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:18:30:848.454 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-01-27 Mon 02:18:30:940.153 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-01-27 Mon 02:18:30:946.288 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668678997760, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-01-27 Mon 02:18:30:946.342 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668678997760, Pid 1]: SANDESH: Send FAILED: 1737944310944517 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-01-27 Mon 02:18:30:946.353 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668678997760, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-01-27 Mon 02:18:30:947.412 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:50680::10.0.0.37:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-01-27 Mon 02:18:30:947.429 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: 10.0.0.22:50680::10.0.0.37:8086(-1) OnSessionEvent TCP Connected 2025-01-27 Mon 02:18:30:947.443 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: 10.0.0.22:50680::10.0.0.37:8086(-1):EvTcpConnected 2025-01-27 Mon 02:18:30:947.478 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1ac96d0 2025-01-27 Mon 02:18:30:947.485 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.496 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.509 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1ac9f50 2025-01-27 Mon 02:18:30:947.515 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.523 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.542 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1aca7d0 2025-01-27 Mon 02:18:30:947.548 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.555 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.565 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1a89240 2025-01-27 Mon 02:18:30:947.570 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.578 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.587 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1a89a00 2025-01-27 Mon 02:18:30:947.592 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.600 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.608 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1a8a210 2025-01-27 Mon 02:18:30:947.613 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.621 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.630 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1acb050 2025-01-27 Mon 02:18:30:947.635 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.643 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.652 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1a8aa90 2025-01-27 Mon 02:18:30:947.657 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.671 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.681 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1a8b310 2025-01-27 Mon 02:18:30:947.686 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.694 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.37 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.702 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1acb8d0 2025-01-27 Mon 02:18:30:947.708 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.715 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.22 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.724 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1ac6c70 2025-01-27 Mon 02:18:30:947.729 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.737 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.746 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1ac74d0 2025-01-27 Mon 02:18:30:947.751 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.758 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.767 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1ac8e50 2025-01-27 Mon 02:18:30:947.773 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.780 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.789 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1ac7d50 2025-01-27 Mon 02:18:30:947.794 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.802 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:947.810 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: RAC_Connect status: 0 0x1ac85d0 2025-01-27 Mon 02:18:30:947.815 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: Connected to REDIS... 2025-01-27 Mon 02:18:30:947.823 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.32 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-01-27 Mon 02:18:30:948.165 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:948.185 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:948.198 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:948.207 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:948.219 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:948.228 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:948.239 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:948.248 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:948.260 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:948.269 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.22:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:948.337 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:948.366 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:948.383 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:948.398 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:948.413 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:948.522 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-01-27 Mon 02:18:30:948.587 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-01-27 Mon 02:18:30:948.613 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: ClientInit 2025-01-27 Mon 02:18:30:948.632 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2660-3.:contrail-query-engine:0:Database count 1 2025-01-27 Mon 02:18:30:948.724 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:18:30:948.733 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:18:30:948.756 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: SANDESH: Send FAILED: 1737944310948327 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-01-27 Mon 02:18:30:948.764 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:18:30:948.773 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:18:30:948.779 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:18:30:948.797 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: SANDESH: Send FAILED: 1737944310948358 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down.10.0.0.32:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:18:30:948.804 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:18:30:948.811 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:18:30:948.818 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:18:30:948.833 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: SANDESH: Send FAILED: 1737944310948607 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down.10.0.0.32:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:18:30:948.844 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:18:30:948.851 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:18:30:948.858 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-01-27 Mon 02:18:30:948.878 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:18:30:948.885 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-01-27 Mon 02:18:30:948.993 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: SANDESH: Send FAILED: 1737944310948715 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737944310825637 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.32:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 31 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 33, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-01-27 Mon 02:18:30:949.007 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-01-27 Mon 02:18:30:949.160 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:949.173 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:949.189 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:949.199 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:949.212 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:949.220 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:949.232 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:949.241 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:949.252 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:949.264 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.32:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:949.297 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:949.323 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:949.338 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:949.352 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:949.365 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:949.400 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:18:30:949.410 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:18:30:949.429 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: SANDESH: Send FAILED: 1737944310949315 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = 10.0.0.37:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:18:30:949.436 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:18:30:953.177 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:953.237 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:953.252 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:953.261 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:953.281 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:953.318 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:953.340 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:953.349 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:953.361 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:953.370 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:953.382 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-01-27 Mon 02:18:30:953.406 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.37:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-01-27 Mon 02:18:30:953.421 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:953.434 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:953.446 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-01-27 Mon 02:18:30:953.522 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-01-27 Mon 02:18:30:953.556 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-01-27 Mon 02:18:30:953.580 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: SANDESH: Send FAILED: 1737944310953301 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-01-27 Mon 02:18:30:953.589 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-01-27 Mon 02:18:30:953.671 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: OnMessage control in state: ClientInit 2025-01-27 Mon 02:18:30:953.690 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-01-27 Mon 02:18:30:953.698 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-01-27 Mon 02:18:30:953.724 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: Received Ctrl Message with size 0 2025-01-27 Mon 02:18:30:953.845 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1737944310825637 collector_name = collector_ip = 10.0.0.37:8086 collector_list= [ [ (*_iter40) = 10.0.0.37:8086, (*_iter40) = 10.0.0.22:8086, (*_iter40) = 10.0.0.32:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 31 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 33, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-01-27 Mon 02:18:30:953.866 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:18:30:953.888 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2660-3. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.37:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.37:9042, (*_iter6) = 10.0.0.32:9042, (*_iter6) = 10.0.0.22:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.37:6379, (*_iter6) = 10.0.0.32:6379, (*_iter6) = 10.0.0.22:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-01-27 Mon 02:18:30:953.899 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-01-27 Mon 02:18:30:953.906 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-01-27 Mon 02:18:30:953.914 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-01-27 Mon 02:18:30:953.937 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: Established 2025-01-27 Mon 02:19:30:826.475 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668678997760, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-01-27 Mon 02:20:30:827.136 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-01-27 Mon 02:21:30:827.790 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-01-27 Mon 02:22:30:828.783 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668791531264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-01-27 Mon 02:23:30:829.330 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668791531264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-01-27 Mon 02:24:30:830.037 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668804126464, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-01-27 Mon 02:25:30:830.661 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-01-27 Mon 02:26:30:831.331 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-01-27 Mon 02:27:30:831.921 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668704188160, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-01-27 Mon 02:28:30:832.479 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668791531264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-01-27 Mon 02:29:30:833.216 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668687394560, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-01-27 Mon 02:30:27:773.056 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8091::10.0.0.22:54036 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:30:30:833.907 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668808324864, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-01-27 Mon 02:31:07:409.266 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8091::10.0.0.22:48894 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:31:28:818.032 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8091::10.0.0.22:54582 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:31:30:836.192 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668683196160, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-01-27 Mon 02:31:53:243.851 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8091::10.0.0.22:53796 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:32:30:837.652 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668795729664, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-01-27 Mon 02:33:06:342.726 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8091::10.0.0.37:41572 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:33:30:723.759 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.37 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-01-27 Mon 02:33:30:724.319 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID 18bf138a-dc57-11ef-9ec2-00000a000025 src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-01-27 Mon 02:33:30:724.381 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QEQueryLog: 18bf138a-dc57-11ef-9ec2-00000a000025 :0:json query is: { end_time: 1737945210710491 , enqueue_time: 1737945210723485 , query_metadata: {"enqueue_time": 1737945210723392} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1737945191023525 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849:ctest-vn22-05230849", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-01-27 Mon 02:33:30:724.621 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-01-27 Mon 02:33:30:724.640 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-01-27 Mon 02:33:30:724.699 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for 18bf138a-dc57-11ef-9ec2-00000a000025 , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-01-27 Mon 02:33:30:724.809 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668699989760, Pid 1]: [SYS_INFO]: QEQueryLog: 18bf138a-dc57-11ef-9ec2-00000a000025 :0:json query is: { end_time: 1737945210710491 , enqueue_time: 1737945210723485 , query_metadata: {"enqueue_time": 1737945210723392} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1737945191023525 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849:ctest-vn22-05230849", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-01-27 Mon 02:33:30:767.046 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:767.099 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:767.108 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:767.131 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:767.139 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:767.146 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:767.154 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:767.177 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:767.185 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:767.192 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:769.535 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:769.572 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:769.582 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:769.591 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:769.599 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:769.615 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:769.623 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:769.630 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:769.638 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 1 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:769.645 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:769.928 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: [SYS_INFO]: QEQueryLog: 18bf138a-dc57-11ef-9ec2-00000a000025 ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849:ctest-vn22-05230849 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-01-27 Mon 02:33:30:783.644 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:783.722 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:783.732 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:783.740 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:783.748 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:783.756 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:783.763 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:783.770 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:783.778 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-01-27 Mon 02:33:30:783.785 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: 18bf138a-dc57-11ef-9ec2-00000a000025 Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-01-27 Mon 02:33:30:783.840 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QEQueryLog: 18bf138a-dc57-11ef-9ec2-00000a000025 :0:json query is: { end_time: 1737945210710491 , enqueue_time: 1737945210723485 , query_metadata: {"enqueue_time": 1737945210723392} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1737945191023525 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849:ctest-vn22-05230849", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-01-27 Mon 02:33:30:784.057 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-01-27 Mon 02:33:30:784.094 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-01-27 Mon 02:33:30:784.142 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-01-27 Mon 02:33:30:784.724 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668787332864, Pid 1]: [SYS_INFO]: QELog: Finished: QID 18bf138a-dc57-11ef-9ec2-00000a000025 Table ObjectRoutingInstance Time(ms) 60 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-01-27 Mon 02:33:30:784.783 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668787332864, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-01-27 Mon 02:33:30:784.824 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668695791360, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2660-3. table = ObjectRoutingInstance query_stats= [ qid = 18bf138a-dc57-11ef-9ec2-00000a000025 where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-99997421:ctest-vn22-05230849:ctest-vn22-05230849", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 41,0, 42,0, 58,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 60 rows = 1 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-01-27 Mon 02:33:30:839.021 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-01-27 Mon 02:34:30:840.214 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668712584960, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-01-27 Mon 02:35:30:841.269 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-01-27 Mon 02:36:30:841.875 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668812523264, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-01-27 Mon 02:37:30:842.458 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-01-27 Mon 02:38:30:842.986 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668691592960, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-01-27 Mon 02:39:30:843.557 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668704188160, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-01-27 Mon 02:40:22:554.672 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8091::10.0.0.37:57896 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-01-27 Mon 02:40:30:844.236 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668708386560, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-01-27 Mon 02:41:30:844.813 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668704188160, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-01-27 Mon 02:41:43:260.093 UTC cn-jenkins-deploy-platform-ansible-os-2660-3 [Thread 139668960385152, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.22:8091::10.0.0.22:42436 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256