2025-03-20 Thu 00:48:35:427.917 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: http-server-port 8091 2025-03-20 Thu 00:48:35:428.005 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Max-tasks 16 2025-03-20 Thu 00:48:35:428.013 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Max-slice 100 2025-03-20 Thu 00:48:35:428.020 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Collectors 10.0.0.42:8086 2025-03-20 Thu 00:48:35:428.025 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Collectors 10.0.0.12:8086 2025-03-20 Thu 00:48:35:428.030 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Collectors 10.0.0.47:8086 2025-03-20 Thu 00:48:35:428.048 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: ROLE : Generator 2025-03-20 Thu 00:48:35:428.055 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-03-20 Thu 00:48:35:428.060 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2923-1. 2025-03-20 Thu 00:48:35:428.065 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: NODE TYPE : Database 2025-03-20 Thu 00:48:35:428.071 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: INSTANCE ID : 0 2025-03-20 Thu 00:48:35:428.076 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-03-20 Thu 00:48:35:428.081 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-03-20 Thu 00:48:35:428.671 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: HTTP Introspect Init 2025-03-20 Thu 00:48:35:428.779 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-03-20 Thu 00:48:35:428.842 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Sandesh Http Server Port: 8091 2025-03-20 Thu 00:48:35:428.883 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-03-20 Thu 00:48:35:428.892 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-03-20 Thu 00:48:35:428.964 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Idle 2025-03-20 Thu 00:48:35:429.116 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: No Client: 1742431715429044 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:48:35:429.296 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-03-20 Thu 00:48:35:429.314 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-03-20 Thu 00:48:35:429.327 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Cassandra Servers: 10.0.0.42:9042 10.0.0.12:9042 10.0.0.47:9042 2025-03-20 Thu 00:48:35:452.172 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvStart in state Idle 2025-03-20 Thu 00:48:35:452.261 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Disconnect 2025-03-20 Thu 00:48:35:452.340 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-03-20 Thu 00:48:35:452.583 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Connect : Start Connect timer 10.0.0.42:8086 2025-03-20 Thu 00:48:35:452.657 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:452.679 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SANDESH: Send FAILED: 1742431715452252 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-03-20 Thu 00:48:35:452.688 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:48:35:452.697 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:452.778 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SANDESH: Send FAILED: 1742431715452329 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:48:35:452.799 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:48:35:452.809 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:452.877 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SANDESH: Send FAILED: 1742431715452386 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:48:35:452.889 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:48:35:452.898 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:452.912 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SANDESH: Send FAILED: 1742431715452406 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-03-20 Thu 00:48:35:452.919 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:48:35:452.927 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:452.988 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SANDESH: Send FAILED: 1742431715452645 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:48:35:452.999 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:48:35:485.234 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-03-20 Thu 00:48:35:485.288 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:48:35:485.376 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:48:35:485.442 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083189118720, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:485.485 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083189118720, Pid 1]: SANDESH: Send FAILED: 1742431715485340 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-03-20 Thu 00:48:35:485.498 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083189118720, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:48:40:499.044 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-03-20 Thu 00:48:40:499.089 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:48:40:499.128 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:48:45:504.074 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-03-20 Thu 00:48:45:504.117 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:48:45:504.139 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:48:50:517.946 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-03-20 Thu 00:48:50:517.992 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:48:50:518.015 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:48:55:554.826 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:48:55:554.873 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:48:55:554.895 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:00:582.016 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:00:582.063 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:00:582.085 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:05:612.602 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:05:612.655 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:05:612.686 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:10:636.127 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:10:636.186 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:10:636.208 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:15:655.074 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:15:655.115 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:15:655.137 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:20:684.264 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:20:684.309 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:20:684.330 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:25:711.594 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:25:711.637 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:25:711.659 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:30:740.950 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:30:741.007 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:35:765.841 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-03-20 Thu 00:49:40:775.840 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:49:40:775.881 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 00:49:40:775.917 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 00:49:40:775.946 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c09620 2025-03-20 Thu 00:49:40:775.961 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:775.974 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:775.904 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: SANDESH: Send FAILED: 1742431780775748 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-03-20 Thu 00:49:40:775.989 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c09cd0 2025-03-20 Thu 00:49:40:776.004 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:40:776.011 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.028 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 00:49:40:776.037 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.047 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 00:49:40:776.061 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c03070 2025-03-20 Thu 00:49:40:776.076 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.085 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.094 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0a0d0 2025-03-20 Thu 00:49:40:776.099 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.107 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.116 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0a840 2025-03-20 Thu 00:49:40:776.122 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.129 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.137 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0b180 2025-03-20 Thu 00:49:40:776.143 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.154 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.163 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0dfc0 2025-03-20 Thu 00:49:40:776.168 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.176 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.185 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0cd40 2025-03-20 Thu 00:49:40:776.191 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.198 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.207 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0e900 2025-03-20 Thu 00:49:40:776.212 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.220 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.228 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0f240 2025-03-20 Thu 00:49:40:776.233 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.241 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.250 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0c400 2025-03-20 Thu 00:49:40:776.255 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.274 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.283 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0bac0 2025-03-20 Thu 00:49:40:776.288 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.296 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.304 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0fb80 2025-03-20 Thu 00:49:40:776.309 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.317 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.326 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c104c0 2025-03-20 Thu 00:49:40:776.331 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.342 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.350 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0d680 2025-03-20 Thu 00:49:40:776.356 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:776.364 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:776.775 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:776.798 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:776.813 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:776.827 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 00:49:40:776.880 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Idle 2025-03-20 Thu 00:49:40:776.947 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:777.061 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: SANDESH: Send FAILED: 1742431780776772 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 51 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 52, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:49:40:777.081 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:49:40:777.091 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:777.111 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: SANDESH: Send FAILED: 1742431780776803 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-03-20 Thu 00:49:40:777.118 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:40:777.127 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 00:49:40:777.171 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:777.275 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: SANDESH: Send FAILED: 1742431780776937 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 55 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 57, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:49:40:777.291 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:49:40:777.373 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:777.415 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: SANDESH: Send FAILED: 1742431780777313 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742431780776879 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742431780777169 ] max_sm_queue_count = 5 ] 2025-03-20 Thu 00:49:40:777.422 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:49:40:777.430 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:777.538 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: SANDESH: Send FAILED: 1742431780777360 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 55 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 57, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 10, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:49:40:777.552 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:49:40:776.833 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:778.401 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:778.414 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:778.428 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:778.438 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:778.458 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:778.468 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:778.536 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:778.567 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:778.617 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SANDESH: Send FAILED: 1742431780778506 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 00:49:40:779.139 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:40:778.568 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:779.151 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:779.188 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:779.191 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SANDESH: Send FAILED: 1742431780778561 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 00:49:40:779.211 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:40:779.213 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:779.246 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:779.274 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:779.284 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:779.298 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:779.307 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:779.320 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:779.329 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:779.341 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:779.350 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:779.362 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:779.371 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:779.383 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:779.392 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:779.404 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:779.413 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:779.425 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:779.434 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:779.449 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:779.458 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:779.472 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:779.480 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:779.520 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:779.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:779.588 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:779.594 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:779.625 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742431780779550 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 00:49:40:779.633 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:40:779.605 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:781.222 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:781.250 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:781.286 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:781.301 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:781.944 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:781.961 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:781.422 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:782.044 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: SANDESH: Send FAILED: 1742431780781271 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 00:49:40:782.069 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:41:776.996 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 00:49:41:777.226 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-03-20 Thu 00:49:41:777.333 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:49:41:777.361 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: SANDESH: Send FAILED: 1742431781777069 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 00:49:41:777.370 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:41:777.379 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:49:41:777.490 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: SANDESH: Send FAILED: 1742431781777323 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:49:41:777.513 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:49:41:779.679 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:56308::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 00:49:41:779.712 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:56308::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 00:49:41:779.721 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:56308::10.0.0.12:8086(-1):EvTcpConnected 2025-03-20 Thu 00:49:41:779.770 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 00:49:41:779.801 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 00:49:41:779.839 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: ClientInit 2025-03-20 Thu 00:49:41:779.860 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-1.:contrail-query-engine:0:Database count 1 2025-03-20 Thu 00:49:41:779.940 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 00:49:41:779.950 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 00:49:41:779.972 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: SANDESH: Send FAILED: 1742431781779832 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 00:49:41:779.980 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 00:49:41:779.989 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 00:49:41:779.995 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 00:49:41:780.007 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 00:49:41:780.014 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 00:49:41:780.115 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: SANDESH: Send FAILED: 1742431781779930 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:49:41:780.141 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 00:49:41:781.537 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: OnMessage control in state: ClientInit 2025-03-20 Thu 00:49:41:781.601 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-03-20 Thu 00:49:41:781.628 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-03-20 Thu 00:49:41:781.658 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Received Ctrl Message with size 0 2025-03-20 Thu 00:49:41:781.802 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742431780776879 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742431780777169 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 5 proxy seq 10 2025-03-20 Thu 00:49:41:781.829 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 00:49:41:781.853 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-03-20 Thu 00:49:41:781.865 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 00:49:41:781.873 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-03-20 Thu 00:49:41:781.897 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 00:49:41:781.931 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Established 2025-03-20 Thu 00:49:50:720.062 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: OnMessage control in state: Established 2025-03-20 Thu 00:49:50:720.123 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-03-20 Thu 00:49:50:720.133 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Established : EvSandeshMessageRecv 2025-03-20 Thu 00:49:50:720.174 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Received Ctrl Message with size 3 2025-03-20 Thu 00:49:50:720.187 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-03-20 Thu 00:49:50:720.198 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-03-20 Thu 00:49:50:720.210 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-03-20 Thu 00:49:50:720.217 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 00:50:10:779.935 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-03-20 Thu 00:51:10:780.799 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-03-20 Thu 00:52:10:782.092 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-03-20 Thu 00:52:38:492.953 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:8091::10.0.0.42:39978 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-03-20 Thu 00:53:03:794.406 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:8091::10.0.0.42:58074 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-03-20 Thu 00:53:10:782.662 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-03-20 Thu 00:54:10:783.763 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-03-20 Thu 00:55:10:784.279 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083214309120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-03-20 Thu 00:56:10:785.300 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083214309120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-03-20 Thu 00:57:10:786.750 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083189118720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-03-20 Thu 00:58:10:787.562 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-03-20 Thu 00:59:10:788.118 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083189118720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-03-20 Thu 01:00:10:789.756 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-03-20 Thu 01:01:10:790.262 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-03-20 Thu 01:02:10:790.909 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-03-20 Thu 01:03:10:792.676 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-03-20 Thu 01:04:10:793.529 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-03-20 Thu 01:05:10:794.099 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-03-20 Thu 01:06:10:795.594 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-03-20 Thu 01:07:10:796.134 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-03-20 Thu 01:08:10:798.396 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-03-20 Thu 01:09:10:799.274 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-03-20 Thu 01:10:10:800.264 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-03-20 Thu 01:11:10:801.112 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-03-20 Thu 01:12:10:802.144 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083210110720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-03-20 Thu 01:13:10:802.635 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083189118720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-03-20 Thu 01:14:10:803.183 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-03-20 Thu 01:15:10:803.673 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-03-20 Thu 01:16:10:804.123 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-03-20 Thu 01:17:10:804.603 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-03-20 Thu 01:18:10:805.103 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-03-20 Thu 01:19:10:805.585 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-03-20 Thu 01:20:10:806.138 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-03-20 Thu 01:21:10:806.637 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-03-20 Thu 01:22:10:807.080 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-03-20 Thu 01:23:10:807.531 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-03-20 Thu 01:24:10:807.990 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-03-20 Thu 01:25:10:808.503 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-03-20 Thu 01:26:10:808.962 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-03-20 Thu 01:27:10:809.494 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-03-20 Thu 01:28:10:810.311 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-03-20 Thu 01:29:10:810.965 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-03-20 Thu 01:30:10:811.529 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-03-20 Thu 01:31:10:811.985 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-03-20 Thu 01:32:10:812.434 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-03-20 Thu 01:33:10:812.886 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-03-20 Thu 01:34:10:813.348 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-03-20 Thu 01:35:10:813.838 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083214309120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-03-20 Thu 01:36:10:814.392 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083214309120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-03-20 Thu 01:37:10:815.082 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082757490432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-03-20 Thu 01:38:10:815.553 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-03-20 Thu 01:39:10:816.018 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-03-20 Thu 01:40:10:816.453 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-03-20 Thu 01:41:10:817.023 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083214309120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-03-20 Thu 01:42:10:817.535 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-03-20 Thu 01:43:10:818.040 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-03-20 Thu 01:44:10:818.535 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-03-20 Thu 01:45:10:818.998 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-03-20 Thu 01:46:10:819.469 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-03-20 Thu 01:47:10:819.978 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-03-20 Thu 01:48:10:820.469 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083214309120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-03-20 Thu 01:49:10:820.897 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 60 without seqno , total = 0 2025-03-20 Thu 01:50:10:821.340 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083214309120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 61 without seqno , total = 0 2025-03-20 Thu 01:51:10:821.851 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 62 without seqno , total = 0 2025-03-20 Thu 01:52:10:827.656 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 63 without seqno , total = 0 2025-03-20 Thu 01:52:36:803.828 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0d680 2025-03-20 Thu 01:52:36:803.885 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:52:36:804.016 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0bac0 2025-03-20 Thu 01:52:36:804.035 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:52:36:804.064 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0c400 2025-03-20 Thu 01:52:36:804.074 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:52:36:804.201 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-03-20 Thu 01:52:36:804.211 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0b180 2025-03-20 Thu 01:52:36:804.235 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:52:36:804.257 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0cd40 2025-03-20 Thu 01:52:36:804.267 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:52:36:804.429 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:52:36:804.525 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:52:36:805.139 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:52:36:805.159 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:52:36:805.172 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:52:36:946.031 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0fb80 2025-03-20 Thu 01:52:36:946.097 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:52:36:946.144 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c104c0 2025-03-20 Thu 01:52:36:946.156 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:52:36:946.357 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0e900 2025-03-20 Thu 01:52:36:946.383 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:52:36:946.411 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-03-20 Thu 01:52:36:946.419 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0dfc0 2025-03-20 Thu 01:52:36:946.427 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:52:36:946.506 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0f240 2025-03-20 Thu 01:52:36:946.534 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:52:36:946.841 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:52:36:946.872 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:52:36:946.899 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:52:36:947.956 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:52:36:947.992 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:52:41:804.520 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:52:41:804.722 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:52:41:805.191 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:52:41:805.262 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:52:41:805.282 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:52:41:805.342 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:52:41:805.387 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:52:41:805.644 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:52:41:805.695 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:52:41:805.822 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:52:41:946.906 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:52:41:947.029 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:52:41:947.064 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:52:41:947.314 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:52:41:947.349 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:52:41:947.363 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:52:41:947.993 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:52:41:948.043 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:52:41:948.248 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:52:41:948.352 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:52:46:805.269 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:52:46:805.385 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:52:46:805.691 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:52:46:805.747 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:52:46:805.875 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:52:46:805.922 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:52:46:805.996 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:52:46:806.064 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:52:46:806.454 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:52:46:819.364 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:52:46:879.213 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.42:56308::10.0.0.12:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-03-20 Thu 01:52:46:879.260 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.42:56308::10.0.0.12:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:52:46:879.280 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:56308::10.0.0.12:8086(-1) OnSessionEvent TCP Connection Closed 2025-03-20 Thu 01:52:46:879.288 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:56308::10.0.0.12:8086(-1):EvTcpClose 2025-03-20 Thu 01:52:46:879.337 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvTcpClose in state Established 2025-03-20 Thu 01:52:46:879.348 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Established : EvTcpClose 2025-03-20 Thu 01:52:46:879.559 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Connect : Start Connect timer 10.0.0.47:8086 2025-03-20 Thu 01:52:46:879.635 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:52:46:879.786 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SANDESH: Send FAILED: 1742435566879427 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = cn-jenkins-deploy-platform-ansible-os-2923-2. collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3072667 calls = 323 average_bytes = 9512 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 64 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 254 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 115, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-03-20 Thu 01:52:46:879.808 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:52:46:879.819 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:52:46:879.841 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SANDESH: Send FAILED: 1742435566879454 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:52:46:879.849 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:52:46:879.858 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-03-20 Thu 01:52:46:879.880 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:52:46:880.004 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SANDESH: Send FAILED: 1742435566879624 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3072667 calls = 323 average_bytes = 9512 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 64 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 254 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 115, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-03-20 Thu 01:52:46:880.021 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:52:46:880.613 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 01:52:46:880.622 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 01:52:46:880.645 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 01:52:46:880.654 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 01:52:46:880.742 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:52:46:880.764 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Idle 2025-03-20 Thu 01:52:46:880.820 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:52:46:880.951 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SANDESH: Send FAILED: 1742435566880709 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3072667 calls = 323 average_bytes = 9512 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 64 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 254 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 6 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 115, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-03-20 Thu 01:52:46:880.967 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:52:46:880.977 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:52:46:880.998 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SANDESH: Send FAILED: 1742435566880728 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:52:46:881.005 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:52:46:881.013 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:52:46:881.028 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:52:46:881.152 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SANDESH: Send FAILED: 1742435566880811 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3072667 calls = 323 average_bytes = 9512 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 64 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 254 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 116, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-03-20 Thu 01:52:46:881.168 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:52:46:947.395 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:52:46:947.541 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:52:46:947.594 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:52:46:947.991 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:52:46:948.054 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:52:46:948.079 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:52:46:948.301 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:52:46:948.387 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:52:46:948.665 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:52:46:948.720 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:52:47:881.422 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:52:47:881.682 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Connect : Start Connect timer 10.0.0.42:8086 2025-03-20 Thu 01:52:47:881.780 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:52:47:881.808 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SANDESH: Send FAILED: 1742435567881493 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:52:47:881.827 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:52:47:881.836 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:52:47:882.012 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SANDESH: Send FAILED: 1742435567881768 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3072667 calls = 323 average_bytes = 9512 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 64 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 254 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 116, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 26, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-03-20 Thu 01:52:47:882.029 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:52:47:883.930 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:43328::10.0.0.42:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:52:47:883.949 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:43328::10.0.0.42:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:52:47:883.956 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:43328::10.0.0.42:8086(-1):EvTcpConnected 2025-03-20 Thu 01:52:47:884.020 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:52:47:884.052 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:52:47:884.089 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit 2025-03-20 Thu 01:52:47:884.111 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-1.:contrail-query-engine:0:Database count 2 2025-03-20 Thu 01:52:47:884.190 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:52:47:884.200 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:52:47:884.233 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SANDESH: Send FAILED: 1742435567884082 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:52:47:884.244 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:52:47:884.256 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:52:47:884.267 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:52:47:884.281 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:52:47:884.291 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:52:47:884.431 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SANDESH: Send FAILED: 1742435567884181 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3072667 calls = 323 average_bytes = 9512 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 64 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 254 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 117, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 324, ] ] ] 2025-03-20 Thu 01:52:47:884.455 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:52:47:886.186 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: OnMessage control in state: ClientInit 2025-03-20 Thu 01:52:47:886.231 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-03-20 Thu 01:52:47:886.241 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-03-20 Thu 01:52:47:886.265 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: Received Ctrl Message with size 0 2025-03-20 Thu 01:52:47:886.469 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3072667 calls = 323 average_bytes = 9512 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 341 dequeues = 341 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 349 dequeues = 349 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Established state_since = 1742431781781930 last_state = Established last_event = scm::EvSandeshSend last_event_at = 1742435556947037 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 321 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 3047303 calls = 320 average_bytes = 9522 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 64 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 108 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 254 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 117, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 324, ] ] max_sm_queue_count = 5 proxy seq 269 2025-03-20 Thu 01:52:47:886.498 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:52:47:886.521 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 19 2025-03-20 Thu 01:52:47:886.532 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:52:47:886.561 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 2 available_connections = 2 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 64 2025-03-20 Thu 01:52:47:886.575 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:52:47:886.582 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 01:52:47:886.609 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: Established 2025-03-20 Thu 01:52:51:806.121 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:52:51:806.245 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:52:51:806.286 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:52:51:806.481 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:52:51:806.547 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:52:51:806.584 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:52:51:806.606 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:52:51:806.765 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:52:51:819.431 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:52:51:819.927 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:52:51:948.075 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:52:51:948.205 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:52:51:948.251 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:52:51:948.571 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:52:51:948.614 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:52:51:948.629 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:52:51:948.720 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:52:51:948.773 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:52:51:949.023 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:52:51:949.068 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:52:56:806.626 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:52:56:806.800 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:52:56:806.850 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:52:56:806.890 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:52:56:807.245 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0d680 2025-03-20 Thu 01:52:56:807.257 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:52:56:807.289 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:52:56:807.381 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0bac0 2025-03-20 Thu 01:52:56:807.394 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:52:56:807.405 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:52:56:807.631 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0b180 2025-03-20 Thu 01:52:56:807.650 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:52:56:807.681 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:52:56:807.719 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:52:56:807.729 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:52:56:807.767 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:52:56:808.258 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:52:56:808.283 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:52:56:808.385 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:52:56:808.403 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:52:56:808.412 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:52:56:808.432 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:52:56:808.457 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0cd40 2025-03-20 Thu 01:52:56:808.464 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:52:56:808.476 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:52:56:809.579 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:52:56:809.602 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:52:56:809.623 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:52:56:819.999 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:52:56:820.517 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0c400 2025-03-20 Thu 01:52:56:820.537 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:52:56:820.567 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:52:56:820.955 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:52:56:820.974 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:52:56:821.006 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:52:56:948.644 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:52:56:948.773 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:52:56:948.813 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:52:56:949.055 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:52:56:949.147 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0fb80 2025-03-20 Thu 01:52:56:949.156 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:52:56:949.178 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:52:56:949.189 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c104c0 2025-03-20 Thu 01:52:56:949.195 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:52:56:949.333 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:52:56:949.347 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0e900 2025-03-20 Thu 01:52:56:949.353 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:52:56:949.360 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:52:56:949.378 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:52:56:949.477 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0dfc0 2025-03-20 Thu 01:52:56:949.485 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:52:56:949.503 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:52:56:949.869 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0f240 2025-03-20 Thu 01:52:56:949.904 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:52:56:949.920 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:52:56:950.001 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:52:56:950.031 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:52:56:950.055 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:52:56:950.070 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:52:56:950.079 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:52:56:950.091 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:52:56:950.100 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:52:56:950.113 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:52:56:950.121 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:52:56:950.214 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:52:56:950.264 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:52:56:950.278 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:52:56:950.296 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:52:56:950.306 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:52:56:950.322 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:10:828.142 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 64 without seqno , total = 0 2025-03-20 Thu 01:54:10:831.228 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 65 without seqno , total = 0 2025-03-20 Thu 01:55:10:831.734 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083189118720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 66 without seqno , total = 0 2025-03-20 Thu 01:55:45:999.452 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0a840 2025-03-20 Thu 01:55:45:999.649 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:001.534 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0a0d0 2025-03-20 Thu 01:55:46:001.575 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:001.622 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c03070 2025-03-20 Thu 01:55:46:001.634 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:001.659 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c09cd0 2025-03-20 Thu 01:55:46:001.669 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:001.697 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-03-20 Thu 01:55:46:001.704 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c09620 2025-03-20 Thu 01:55:46:001.720 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:011.949 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:43328::10.0.0.42:8086(-1) OnSessionEvent TCP Connection Closed 2025-03-20 Thu 01:55:46:011.973 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:43328::10.0.0.42:8086(-1):EvTcpClose 2025-03-20 Thu 01:55:46:011.998 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:55:46:012.022 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:55:46:012.035 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:55:46:012.048 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:55:46:012.061 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:55:46:012.105 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Processing scm::EvTcpClose in state Established 2025-03-20 Thu 01:55:46:012.116 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Established : EvTcpClose 2025-03-20 Thu 01:55:46:012.366 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-03-20 Thu 01:55:46:012.462 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:012.612 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SANDESH: Send FAILED: 1742435746012202 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 2 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = cn-jenkins-deploy-platform-ansible-os-2923-1. collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3274859 calls = 344 average_bytes = 9519 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 68 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 154 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 268 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 163, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 348, ] ] ] 2025-03-20 Thu 01:55:46:012.875 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:012.898 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:012.936 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SANDESH: Send FAILED: 1742435746012238 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.42:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:55:46:012.947 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:46:012.959 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-03-20 Thu 01:55:46:012.992 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:013.134 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SANDESH: Send FAILED: 1742435746012449 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3274859 calls = 344 average_bytes = 9519 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 68 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 154 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 268 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 163, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 348, ] ] ] 2025-03-20 Thu 01:55:46:013.152 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:022.009 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:47906::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:55:46:022.034 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:47906::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:55:46:022.042 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:47906::10.0.0.12:8086(-1):EvTcpConnected 2025-03-20 Thu 01:55:46:022.094 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:55:46:022.125 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:55:46:022.165 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit 2025-03-20 Thu 01:55:46:022.188 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-1.:contrail-query-engine:0:Database count 3 2025-03-20 Thu 01:55:46:022.291 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:022.301 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:55:46:022.325 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SANDESH: Send FAILED: 1742435746022157 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.42:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:55:46:022.333 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:55:46:022.342 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:022.348 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:55:46:022.359 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:022.373 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:022.503 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SANDESH: Send FAILED: 1742435746022280 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3274859 calls = 344 average_bytes = 9519 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 68 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 154 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 268 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 164, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 348, ] ] ] 2025-03-20 Thu 01:55:46:022.518 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:112.057 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: OnMessage control in state: ClientInit 2025-03-20 Thu 01:55:46:142.681 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-03-20 Thu 01:55:46:142.717 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-03-20 Thu 01:55:46:142.758 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Received Ctrl Message with size 0 2025-03-20 Thu 01:55:46:142.977 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 3464 calls = 6 average_bytes = 577 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3274859 calls = 344 average_bytes = 9519 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 375 dequeues = 375 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 390 dequeues = 390 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Established state_since = 1742435567886607 last_state = Established last_event = scm::EvSandeshSend last_event_at = 1742435710831782 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 21 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 176500 calls = 18 average_bytes = 9805 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 68 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 154 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 268 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 164, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 33, _iter59->first = sent _iter59->second = 348, ] ] max_sm_queue_count = 6 proxy seq 285 2025-03-20 Thu 01:55:46:143.023 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:55:46:143.050 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.42:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 26 2025-03-20 Thu 01:55:46:143.065 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:55:46:143.099 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 67 2025-03-20 Thu 01:55:46:143.108 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:55:46:143.116 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 01:55:46:143.148 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Established 2025-03-20 Thu 01:55:46:174.030 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0c400 2025-03-20 Thu 01:55:46:174.078 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:174.116 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0cd40 2025-03-20 Thu 01:55:46:174.128 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:174.170 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-03-20 Thu 01:55:46:174.179 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0b180 2025-03-20 Thu 01:55:46:174.188 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:174.232 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0bac0 2025-03-20 Thu 01:55:46:174.242 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:174.259 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0d680 2025-03-20 Thu 01:55:46:174.279 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:174.296 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0f240 2025-03-20 Thu 01:55:46:174.305 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:174.327 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-03-20 Thu 01:55:46:174.334 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0dfc0 2025-03-20 Thu 01:55:46:174.342 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:174.364 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0e900 2025-03-20 Thu 01:55:46:174.373 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:174.390 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c0fb80 2025-03-20 Thu 01:55:46:174.399 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:174.429 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Disconnect status: -1 0x2c104c0 2025-03-20 Thu 01:55:46:174.439 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:174.942 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:55:46:174.968 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:55:46:175.033 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:55:46:175.073 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:55:46:175.096 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:55:46:175.162 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:55:46:175.175 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:55:46:175.193 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:55:46:175.206 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:55:46:178.723 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:55:46:228.481 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.42:47906::10.0.0.12:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:55:46:228.527 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:47906::10.0.0.12:8086(-1) OnSessionEvent TCP Connection Closed 2025-03-20 Thu 01:55:46:228.536 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:47906::10.0.0.12:8086(-1):EvTcpClose 2025-03-20 Thu 01:55:46:228.586 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvTcpClose in state Established 2025-03-20 Thu 01:55:46:228.598 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Established : EvTcpClose 2025-03-20 Thu 01:55:46:228.815 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Connect : Start Connect timer 10.0.0.47:8086 2025-03-20 Thu 01:55:46:228.907 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:229.086 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435746228679 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 3 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = cn-jenkins-deploy-platform-ansible-os-2923-2. collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3328558 calls = 349 average_bytes = 9537 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 177, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 356, ] ] ] 2025-03-20 Thu 01:55:46:229.106 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:229.118 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:229.145 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435746228708 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:46:229.152 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:46:229.162 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-03-20 Thu 01:55:46:229.189 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:229.328 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435746228890 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3328558 calls = 349 average_bytes = 9537 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 177, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 356, ] ] ] 2025-03-20 Thu 01:55:46:229.344 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:232.100 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:37636::10.0.0.47:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:55:46:232.135 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:37636::10.0.0.47:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:55:46:232.143 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:37636::10.0.0.47:8086(-1):EvTcpConnected 2025-03-20 Thu 01:55:46:233.282 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:55:46:233.300 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:55:46:233.359 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit 2025-03-20 Thu 01:55:46:233.376 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-1.:contrail-query-engine:0:Database count 4 2025-03-20 Thu 01:55:46:233.503 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:233.513 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:55:46:233.536 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SANDESH: Send FAILED: 1742435746233351 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:46:233.544 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:55:46:233.553 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:233.560 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:55:46:233.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:233.576 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:233.719 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SANDESH: Send FAILED: 1742435746233484 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3328558 calls = 349 average_bytes = 9537 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 38, _iter59->first = sent _iter59->second = 356, ] ] ] 2025-03-20 Thu 01:55:46:233.739 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:359.600 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:37636::10.0.0.47:8086(-1) OnSessionEvent TCP Connection Closed 2025-03-20 Thu 01:55:46:359.635 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:37636::10.0.0.47:8086(-1):EvTcpClose 2025-03-20 Thu 01:55:46:359.696 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-03-20 Thu 01:55:46:359.707 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvTcpClose 2025-03-20 Thu 01:55:46:359.824 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Return to idle with 1000 2025-03-20 Thu 01:55:46:359.833 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Idle 2025-03-20 Thu 01:55:46:359.890 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:46:360.034 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SANDESH: Send FAILED: 1742435746359785 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3330131 calls = 350 average_bytes = 9514 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 357, ] ] ] 2025-03-20 Thu 01:55:46:360.064 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:360.080 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:46:360.107 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SANDESH: Send FAILED: 1742435746359813 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:46:360.114 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:46:360.123 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:55:46:360.152 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:46:360.290 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SANDESH: Send FAILED: 1742435746359880 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3330131 calls = 350 average_bytes = 9514 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 40, _iter59->first = sent _iter59->second = 357, ] ] ] 2025-03-20 Thu 01:55:46:360.307 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:47:361.958 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:55:47:362.186 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Connect : Start Connect timer 10.0.0.42:8086 2025-03-20 Thu 01:55:47:362.304 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:47:362.333 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: SANDESH: Send FAILED: 1742435747362026 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:47:362.342 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:47:362.351 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:47:362.483 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: SANDESH: Send FAILED: 1742435747362290 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3330131 calls = 350 average_bytes = 9514 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 178, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 43, _iter59->first = sent _iter59->second = 357, ] ] ] 2025-03-20 Thu 01:55:47:362.506 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:47:364.097 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:51738::10.0.0.42:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:55:47:364.125 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:51738::10.0.0.42:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:55:47:364.134 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:51738::10.0.0.42:8086(-1):EvTcpConnected 2025-03-20 Thu 01:55:47:364.671 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:55:47:364.682 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:55:47:364.714 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: ClientInit 2025-03-20 Thu 01:55:47:364.730 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-1.:contrail-query-engine:0:Database count 5 2025-03-20 Thu 01:55:47:364.806 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:47:364.815 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:55:47:364.845 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: SANDESH: Send FAILED: 1742435747364707 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:47:364.853 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:55:47:364.861 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:47:364.868 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:55:47:364.879 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:47:364.885 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:47:365.013 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: SANDESH: Send FAILED: 1742435747364796 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3330131 calls = 350 average_bytes = 9514 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 179, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 45, _iter59->first = sent _iter59->second = 357, ] ] ] 2025-03-20 Thu 01:55:47:365.030 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:47:374.724 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:51738::10.0.0.42:8086(-1) OnSessionEvent TCP Connection Closed 2025-03-20 Thu 01:55:47:374.761 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:51738::10.0.0.42:8086(-1):EvTcpClose 2025-03-20 Thu 01:55:47:375.803 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-03-20 Thu 01:55:47:375.834 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: ClientInit : EvTcpClose 2025-03-20 Thu 01:55:47:376.029 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Return to idle with 1000 2025-03-20 Thu 01:55:47:376.045 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Idle 2025-03-20 Thu 01:55:47:376.142 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:47:376.416 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SANDESH: Send FAILED: 1742435747375951 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3331704 calls = 351 average_bytes = 9492 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 179, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 47, _iter59->first = sent _iter59->second = 358, ] ] ] 2025-03-20 Thu 01:55:47:377.488 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:47:377.518 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:47:377.558 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SANDESH: Send FAILED: 1742435747375992 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:47:377.571 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:47:377.586 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:55:47:377.601 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:47:377.855 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SANDESH: Send FAILED: 1742435747376126 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3331704 calls = 351 average_bytes = 9492 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 179, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 47, _iter59->first = sent _iter59->second = 358, ] ] ] 2025-03-20 Thu 01:55:47:377.908 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:48:376.836 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:55:48:377.041 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-03-20 Thu 01:55:48:377.137 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:48:377.164 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SANDESH: Send FAILED: 1742435748376907 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:48:377.173 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:48:377.182 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:48:377.334 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SANDESH: Send FAILED: 1742435748377125 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3331704 calls = 351 average_bytes = 9492 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 179, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 50, _iter59->first = sent _iter59->second = 358, ] ] ] 2025-03-20 Thu 01:55:48:377.356 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:48:379.532 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:48018::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:55:48:379.557 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:48018::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:55:48:379.565 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:48018::10.0.0.12:8086(-1):EvTcpConnected 2025-03-20 Thu 01:55:48:379.607 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:55:48:379.617 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:55:48:379.648 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit 2025-03-20 Thu 01:55:48:379.666 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-1.:contrail-query-engine:0:Database count 6 2025-03-20 Thu 01:55:48:379.745 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:48:379.754 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:55:48:379.776 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SANDESH: Send FAILED: 1742435748379642 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:48:379.783 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:55:48:379.792 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:48:379.799 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:55:48:379.809 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:48:379.815 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:48:379.950 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: SANDESH: Send FAILED: 1742435748379735 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3331704 calls = 351 average_bytes = 9492 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 180, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 358, ] ] ] 2025-03-20 Thu 01:55:48:379.963 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083205912320, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:48:380.963 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:48018::10.0.0.12:8086(-1) OnSessionEvent TCP Connection Closed 2025-03-20 Thu 01:55:48:380.987 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:48018::10.0.0.12:8086(-1):EvTcpClose 2025-03-20 Thu 01:55:48:381.029 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-03-20 Thu 01:55:48:381.039 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: ClientInit : EvTcpClose 2025-03-20 Thu 01:55:48:381.164 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Return to idle with 1000 2025-03-20 Thu 01:55:48:381.174 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Idle 2025-03-20 Thu 01:55:48:381.242 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:48:381.390 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SANDESH: Send FAILED: 1742435748381107 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3333277 calls = 352 average_bytes = 9469 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 180, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 54, _iter59->first = sent _iter59->second = 359, ] ] ] 2025-03-20 Thu 01:55:48:381.410 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:48:381.421 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:48:381.443 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SANDESH: Send FAILED: 1742435748381131 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:48:381.450 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:48:381.459 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:55:48:381.481 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:48:381.656 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: SANDESH: Send FAILED: 1742435748381232 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3333277 calls = 352 average_bytes = 9469 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 180, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 54, _iter59->first = sent _iter59->second = 359, ] ] ] 2025-03-20 Thu 01:55:48:381.671 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082765883136, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:49:381.292 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:55:49:381.503 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Connect : Start Connect timer 10.0.0.47:8086 2025-03-20 Thu 01:55:49:381.602 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:49:381.630 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SANDESH: Send FAILED: 1742435749381361 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:49:381.650 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:49:381.659 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:49:381.796 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SANDESH: Send FAILED: 1742435749381590 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 6 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3333277 calls = 352 average_bytes = 9469 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 180, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 359, ] ] ] 2025-03-20 Thu 01:55:49:381.813 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:49:385.170 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:37656::10.0.0.47:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:55:49:385.200 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:37656::10.0.0.47:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:55:49:385.208 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:37656::10.0.0.47:8086(-1):EvTcpConnected 2025-03-20 Thu 01:55:49:387.805 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:55:49:387.830 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:55:49:387.877 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: ClientInit 2025-03-20 Thu 01:55:49:387.895 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-1.:contrail-query-engine:0:Database count 7 2025-03-20 Thu 01:55:49:387.973 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:49:387.982 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:55:49:388.006 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435749387869 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:49:388.013 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:55:49:388.022 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:49:388.029 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:55:49:388.040 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:49:388.046 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:49:388.179 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435749387963 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3333277 calls = 352 average_bytes = 9469 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 59, _iter59->first = sent _iter59->second = 359, ] ] ] 2025-03-20 Thu 01:55:49:388.197 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:51:012.063 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:55:51:012.193 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:55:51:012.251 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:55:51:012.278 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:55:51:012.289 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:55:51:012.317 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:55:51:012.343 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:55:51:012.375 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:55:51:012.390 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:55:51:012.402 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:55:51:175.007 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:55:51:175.161 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:55:51:175.228 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:55:51:175.267 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:55:51:175.297 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:55:51:175.326 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:55:51:175.354 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:55:51:175.384 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:55:51:175.416 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:55:51:175.614 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:55:51:175.695 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:55:51:175.716 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:55:51:175.730 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:55:51:175.742 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:55:51:175.755 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:55:51:175.768 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:55:51:175.780 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:55:51:175.793 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:55:51:178.789 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:55:51:179.175 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:55:56:012.312 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:55:56:012.461 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:55:56:012.517 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:55:56:012.551 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:55:56:012.565 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:55:56:012.598 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:55:56:012.624 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:55:56:012.657 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:55:56:012.671 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:55:56:012.683 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:55:56:175.714 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:55:56:175.839 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:55:56:175.880 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:55:56:175.911 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:55:56:175.951 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:55:56:175.995 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:55:56:176.027 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:55:56:176.088 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:55:56:176.137 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:55:56:176.196 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:55:56:176.237 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:55:56:176.259 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:55:56:176.278 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:55:56:176.298 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:55:56:176.315 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:55:56:176.453 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:55:56:176.511 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:55:56:179.114 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:55:56:179.226 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:55:56:179.656 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:55:57:388.290 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: 10.0.0.47:8086 EvConnectTimerExpired in state ClientInit 2025-03-20 Thu 01:55:57:388.369 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2025-03-20 Thu 01:55:57:388.385 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: ClientInit : EvConnectTimerExpired 2025-03-20 Thu 01:55:57:388.623 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Return to idle with 1000 2025-03-20 Thu 01:55:57:388.646 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Idle 2025-03-20 Thu 01:55:57:388.743 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:57:388.974 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SANDESH: Send FAILED: 1742435757388496 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 7 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3334850 calls = 353 average_bytes = 9447 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 61, _iter59->first = sent _iter59->second = 360, ] ] ] 2025-03-20 Thu 01:55:57:389.021 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:57:389.038 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:57:389.074 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SANDESH: Send FAILED: 1742435757388536 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:57:389.086 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:57:389.099 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:55:57:389.135 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:57:389.361 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: SANDESH: Send FAILED: 1742435757388728 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 7 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3334850 calls = 353 average_bytes = 9447 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 61, _iter59->first = sent _iter59->second = 360, ] ] ] 2025-03-20 Thu 01:55:57:389.392 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082715518720, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:58:388.745 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:55:58:388.999 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: Connect : Start Connect timer 10.0.0.42:8086 2025-03-20 Thu 01:55:58:389.107 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:58:389.136 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SANDESH: Send FAILED: 1742435758388822 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:58:389.145 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:58:389.155 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:58:389.370 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: SANDESH: Send FAILED: 1742435758389093 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 7 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3334850 calls = 353 average_bytes = 9447 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 14 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 181, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 64, _iter59->first = sent _iter59->second = 360, ] ] ] 2025-03-20 Thu 01:55:58:389.407 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082749097728, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:58:391.263 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:60530::10.0.0.42:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:55:58:391.291 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:60530::10.0.0.42:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:55:58:391.300 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:60530::10.0.0.42:8086(-1):EvTcpConnected 2025-03-20 Thu 01:55:58:391.355 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:55:58:391.385 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:55:58:391.424 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: ClientInit 2025-03-20 Thu 01:55:58:391.440 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-1.:contrail-query-engine:0:Database count 8 2025-03-20 Thu 01:55:58:391.520 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:58:391.541 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:55:58:391.565 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: SANDESH: Send FAILED: 1742435758391416 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:58:391.573 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:55:58:391.583 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:58:391.590 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:55:58:391.602 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:58:391.609 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:58:391.772 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: SANDESH: Send FAILED: 1742435758391509 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3334850 calls = 353 average_bytes = 9447 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 7 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 15 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 182, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 66, _iter59->first = sent _iter59->second = 360, ] ] ] 2025-03-20 Thu 01:55:58:391.792 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083201713920, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:58:653.993 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.42:60530::10.0.0.42:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-03-20 Thu 01:55:58:654.035 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.42:60530::10.0.0.42:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:55:58:654.056 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:60530::10.0.0.42:8086(-1) OnSessionEvent TCP Connection Closed 2025-03-20 Thu 01:55:58:654.065 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:60530::10.0.0.42:8086(-1):EvTcpClose 2025-03-20 Thu 01:55:58:654.696 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-03-20 Thu 01:55:58:654.708 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: ClientInit : EvTcpClose 2025-03-20 Thu 01:55:58:654.838 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Return to idle with 1000 2025-03-20 Thu 01:55:58:654.849 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Idle 2025-03-20 Thu 01:55:58:654.906 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:58:655.057 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435758654786 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:55:58:655.084 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:58:655.094 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:58:655.116 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435758654813 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:58:655.124 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:58:655.132 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:55:58:655.147 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:58:655.277 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435758654896 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 68, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:55:58:655.293 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:59:655.595 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:55:59:655.942 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-03-20 Thu 01:55:59:656.046 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:59:656.079 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435759655800 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:59:656.088 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:59:656.099 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:59:656.262 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435759656030 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 39 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 71, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:55:59:656.295 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:59:656.910 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 01:55:59:656.923 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 01:55:59:657.476 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 01:55:59:657.487 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 01:55:59:657.602 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:55:59:657.636 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Idle 2025-03-20 Thu 01:55:59:657.696 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:59:657.852 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435759657557 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 17 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 184, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:55:59:657.872 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:59:657.904 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:59:657.927 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435759657580 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:59:657.934 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:59:657.943 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:55:59:657.952 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:59:658.089 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: SANDESH: Send FAILED: 1742435759657685 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 185, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:55:59:658.106 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083193317120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:01:012.584 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:01:012.789 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:01:012.828 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:01:012.881 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:01:012.928 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:01:012.972 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:01:013.024 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:01:013.050 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:01:013.072 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:01:013.091 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:01:176.266 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:01:176.398 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:01:176.435 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:01:176.462 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:01:176.496 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:01:176.522 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:01:176.563 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:01:176.591 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:01:176.735 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:01:176.764 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:01:176.778 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:01:176.824 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:01:176.838 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:01:176.970 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:01:177.083 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:01:177.182 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:01:179.168 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:01:179.517 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:01:179.681 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:01:180.057 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:06:012.861 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:06:013.015 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:06:013.056 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:06:013.108 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:06:013.122 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:06:013.150 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:06:013.187 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:06:013.202 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:06:013.223 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:06:013.256 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:06:176.798 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:06:176.949 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:06:177.007 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:06:177.032 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:06:177.055 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:06:177.170 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:06:177.226 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:06:177.252 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:06:177.711 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:06:177.802 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:06:177.822 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:06:177.838 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:06:177.854 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:06:177.985 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:06:178.018 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:06:178.043 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:06:179.568 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:06:180.094 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:06:180.126 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:06:180.529 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:10:833.336 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 67 without seqno , total = 0 2025-03-20 Thu 01:56:10:833.393 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:10:833.459 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SANDESH: Send FAILED: 1742435770833203 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 438 dequeues = 438 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 11 dequeues = 11 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 481 dequeues = 481 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742435759657634 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742435759657951 ] max_sm_queue_count = 6 ] 2025-03-20 Thu 01:56:10:833.482 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:10:833.491 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:10:833.629 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SANDESH: Send FAILED: 1742435770833317 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 185, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 76, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:56:10:833.643 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:11:013.151 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:11:013.438 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:11:013.506 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:11:013.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:11:013.619 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:11:013.664 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:11:013.718 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:11:013.744 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:11:013.763 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:11:013.781 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:11:177.851 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:11:178.093 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:11:178.142 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:11:178.175 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:11:178.245 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:11:178.281 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:11:178.321 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:11:178.369 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:11:178.445 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:11:178.492 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:11:178.518 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:11:178.540 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:11:178.560 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:11:178.952 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:11:179.032 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:11:179.050 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:11:180.148 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:11:180.601 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:11:180.686 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:11:181.014 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:16:013.533 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:16:013.687 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:16:013.778 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:16:013.820 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:16:013.859 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:16:013.876 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:16:013.916 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:16:013.947 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:16:013.980 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:16:014.000 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:16:178.522 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:16:178.676 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:16:178.710 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:16:178.743 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:16:178.772 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:16:179.063 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:16:179.119 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:16:179.134 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:16:179.146 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:16:179.225 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:16:179.242 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:16:179.253 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:16:179.285 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:16:179.493 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:16:179.566 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:16:179.601 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:16:180.719 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:16:181.046 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:16:181.155 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:16:181.418 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:21:013.748 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:21:013.930 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:21:013.967 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:21:014.034 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:21:014.054 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:21:014.089 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:21:014.118 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:21:014.152 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:21:014.167 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:21:014.182 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:21:179.821 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:21:179.945 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:21:179.994 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:21:180.033 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:21:180.061 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:21:180.089 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:21:180.117 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:21:180.145 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:21:180.263 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:21:180.326 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:21:180.342 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:21:180.358 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:21:180.372 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:21:180.523 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:21:180.543 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:21:181.192 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:21:181.448 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:21:181.653 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:21:181.756 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:21:197.962 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:26:013.996 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:26:014.174 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:26:014.201 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:26:014.250 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:26:014.263 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:26:014.291 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:26:014.319 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:26:014.351 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:26:014.365 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:26:014.377 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:26:180.458 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:26:180.619 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:26:180.671 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:26:180.709 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:26:180.742 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:26:180.791 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:26:180.824 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:26:180.886 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:26:180.932 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:26:180.947 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:26:180.967 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:26:181.049 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:26:181.134 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:26:181.263 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:26:181.688 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:26:181.798 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:26:182.045 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:26:182.085 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:26:198.098 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:26:198.789 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:29:658.377 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:56:29:658.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Connect : Start Connect timer 10.0.0.47:8086 2025-03-20 Thu 01:56:29:658.676 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:56:29:658.703 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435789658449 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:29:658.713 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:29:658.733 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:56:29:658.924 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435789658662 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 34 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 185, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 78, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:56:29:658.947 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:29:660.297 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 01:56:29:660.314 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 01:56:29:661.611 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 01:56:29:661.641 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 01:56:29:661.767 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:56:29:661.800 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Idle 2025-03-20 Thu 01:56:29:661.864 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:29:662.023 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SANDESH: Send FAILED: 1742435789661717 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 18 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 185, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 80, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:56:29:662.052 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:29:662.063 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:29:662.086 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SANDESH: Send FAILED: 1742435789661746 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:29:662.094 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:29:662.103 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:56:29:662.130 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:29:662.274 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SANDESH: Send FAILED: 1742435789661854 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 80, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:56:29:662.289 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:31:014.242 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:31:014.398 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:31:014.423 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:31:014.456 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:31:014.487 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:31:014.522 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:31:014.537 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:31:014.549 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:31:014.568 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:31:014.600 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:31:180.975 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:31:181.117 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:31:181.156 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:31:181.199 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:31:181.238 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:31:181.266 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:31:181.307 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:31:181.426 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:31:181.458 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:31:181.482 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:31:181.496 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:31:181.508 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:31:181.639 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:31:181.720 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:31:182.071 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:31:182.126 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:31:182.486 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:31:182.526 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:31:198.867 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:31:199.470 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:36:014.457 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:36:014.601 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:36:014.638 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:36:014.672 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:36:014.712 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:36:014.748 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:36:014.764 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:36:014.776 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:36:014.785 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:36:014.815 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:36:181.492 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:36:181.612 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:36:181.668 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:36:181.714 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:36:181.749 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:36:181.780 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:36:181.828 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:36:181.866 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:36:181.908 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:36:181.929 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:36:181.942 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:36:181.960 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:36:182.143 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:36:182.288 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:36:182.521 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:36:182.574 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:36:182.939 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:36:182.989 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:36:199.536 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:36:200.144 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:40:834.989 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:40:835.082 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SANDESH: Send FAILED: 1742435800834869 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 445 dequeues = 445 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 491 dequeues = 491 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742435789661799 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742435789662129 ] max_sm_queue_count = 6 ] 2025-03-20 Thu 01:56:40:835.096 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:40:835.105 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:40:835.249 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SANDESH: Send FAILED: 1742435800834964 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 83, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:56:40:835.275 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:41:014.706 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:41:014.948 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:41:014.997 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:41:015.060 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:41:015.114 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:41:015.164 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:41:015.227 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:41:015.249 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:41:015.263 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:41:015.275 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:41:181.961 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:41:182.092 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:41:182.159 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:41:182.197 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:41:182.245 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:41:182.288 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:41:182.338 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:41:182.389 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:41:182.420 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:41:182.439 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:41:182.458 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:41:182.486 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:41:182.626 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:41:182.714 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:41:183.078 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:41:183.136 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:41:183.439 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:41:183.464 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:41:200.229 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:41:200.851 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:45:821.410 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:45:821.487 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: SANDESH: Send FAILED: 1742435805821373 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-03-20 Thu 01:56:45:821.499 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083197515520, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-03-20 Thu 01:56:46:015.016 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:46:015.147 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:46:015.262 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:46:015.308 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:46:015.322 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:46:015.350 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:46:015.378 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:46:015.410 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:46:015.425 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:46:015.437 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:46:182.454 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:46:182.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:46:182.604 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:46:182.642 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:46:182.669 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:46:182.702 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:46:182.742 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:46:182.867 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:46:182.926 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:46:182.945 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:46:182.959 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:46:182.972 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:46:183.125 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:46:183.482 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:46:183.543 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:46:183.631 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:46:183.895 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:46:183.927 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:46:200.914 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:46:201.494 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:51:015.222 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:51:015.367 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:51:015.392 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:51:015.440 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:51:015.458 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:51:015.497 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:51:015.534 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:51:015.550 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:51:015.560 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:51:015.591 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:51:182.955 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:51:183.100 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:51:183.144 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:51:183.172 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:51:183.198 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:51:183.238 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:51:183.405 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:51:183.530 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:51:183.568 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:51:183.587 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:51:183.605 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:51:183.623 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:51:183.655 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:51:183.975 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:51:184.022 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:51:184.052 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:51:184.253 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:51:184.364 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:51:201.555 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:51:202.120 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:56:016.370 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:56:56:016.508 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:56:56:016.545 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:56:56:016.573 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:56:56:016.600 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:56:56:016.640 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:56:56:016.663 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:56:56:016.676 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:56:56:016.688 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:56:56:016.700 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:56:56:186.832 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:56:56:186.968 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:56:56:187.001 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:56:56:187.029 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:56:56:187.055 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:56:56:187.083 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:56:56:187.111 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:56:56:187.141 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:56:56:187.169 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:56:56:187.659 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:56:56:187.711 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:56:56:187.728 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:56:56:187.741 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:56:56:187.755 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:56:56:187.768 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:56:56:187.782 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:56:56:187.795 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:56:56:187.817 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:56:56:202.191 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:56:56:202.740 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:56:59:661.950 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:56:59:662.142 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Connect : Start Connect timer 10.0.0.42:8086 2025-03-20 Thu 01:56:59:662.218 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 01:56:59:662.249 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 01:56:59:662.254 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:56:59:662.286 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435819662019 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:59:662.295 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:59:662.304 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:56:59:662.447 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435819662242 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 36 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 86, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:56:59:662.471 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:59:662.481 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 01:56:59:662.489 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 01:56:59:662.583 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:56:59:662.611 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Idle 2025-03-20 Thu 01:56:59:662.667 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:59:662.806 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435819662540 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 19 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 186, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 88, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:56:59:662.823 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:59:662.833 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:59:662.854 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435819662560 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:59:662.861 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:59:662.869 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:56:59:662.887 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:59:663.017 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435819662659 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 50 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 187, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 88, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:56:59:663.043 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:01:016.710 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:57:01:016.827 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:57:01:016.879 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:57:01:016.904 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:57:01:016.915 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:57:01:016.955 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:57:01:016.983 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:57:01:017.015 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:57:01:017.030 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:57:01:017.042 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:57:01:188.016 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:57:01:188.137 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:57:01:188.169 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:57:01:188.204 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:57:01:188.243 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:57:01:188.272 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:57:01:188.303 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:57:01:188.331 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:57:01:188.359 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:57:01:188.619 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:57:01:188.665 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:57:01:188.711 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:57:01:188.728 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:57:01:188.742 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:57:01:188.755 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:57:01:188.768 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:57:01:188.839 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:57:01:188.965 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:57:01:202.803 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:57:01:203.362 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:57:06:016.945 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:57:06:017.048 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:57:06:017.086 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:57:06:017.112 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:57:06:017.137 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:57:06:017.249 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:57:06:017.274 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:57:06:017.311 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:57:06:017.344 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:57:06:017.358 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:57:06:188.698 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:57:06:188.828 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:57:06:188.867 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:57:06:188.896 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:57:06:188.923 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:57:06:188.959 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:57:06:188.993 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:57:06:189.020 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:57:06:189.067 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:57:06:189.114 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:57:06:189.155 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:57:06:189.176 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:57:06:189.189 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:57:06:189.233 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:57:06:189.255 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:57:06:189.332 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:57:06:189.393 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:57:06:189.407 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:57:06:203.422 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:57:06:204.351 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:57:10:835.180 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 68 without seqno , total = 0 2025-03-20 Thu 01:57:10:835.239 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:10:835.305 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435830835073 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 453 dequeues = 453 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 13 dequeues = 13 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 502 dequeues = 502 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742435819662609 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742435819662885 ] max_sm_queue_count = 6 ] 2025-03-20 Thu 01:57:10:835.329 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:10:835.338 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:10:835.474 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435830835165 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 38 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 166 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 52 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 187, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 91, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:57:10:835.488 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:11:017.317 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:57:11:017.457 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:57:11:017.525 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:57:11:017.553 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:57:11:017.566 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:57:11:017.602 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:57:11:017.648 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:57:11:017.691 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:57:11:017.706 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:57:11:017.721 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:57:11:189.571 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:57:11:189.687 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:57:11:189.719 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:57:11:189.747 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:57:11:189.773 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:57:11:189.802 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:57:11:189.829 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:57:11:189.855 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:57:11:189.912 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:57:11:190.009 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:57:11:190.038 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:57:11:190.054 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:57:11:190.149 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:57:11:190.165 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:57:11:190.178 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:57:11:190.192 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:57:11:190.205 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:57:11:190.370 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:57:11:205.942 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:57:11:206.729 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:57:16:017.593 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:57:16:017.734 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:57:16:017.788 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:57:16:017.812 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:57:16:017.823 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:57:16:017.853 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:57:16:017.893 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:57:16:017.928 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:57:16:017.944 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:57:16:017.956 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:57:16:190.068 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:57:16:190.228 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:57:16:190.266 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:57:16:190.306 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:57:16:190.334 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:57:16:190.361 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:57:16:190.390 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:57:16:190.420 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:57:16:190.447 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:57:16:190.592 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:57:16:190.648 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:57:16:190.665 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:57:16:190.681 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:57:16:190.694 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:57:16:190.721 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:57:16:190.822 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:57:16:190.908 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:57:16:190.933 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:57:16:206.802 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:57:16:207.510 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:57:21:017.844 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:57:21:017.981 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:57:21:018.033 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a840 2025-03-20 Thu 01:57:21:018.059 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0a0d0 2025-03-20 Thu 01:57:21:018.070 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:57:21:018.099 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:57:21:018.124 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:57:21:018.157 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c03070 2025-03-20 Thu 01:57:21:018.171 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09cd0 2025-03-20 Thu 01:57:21:018.183 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c09620 2025-03-20 Thu 01:57:21:191.726 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:57:21:191.833 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:57:21:191.867 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:57:21:191.897 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:57:21:191.924 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:57:21:191.953 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:57:21:191.981 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:57:21:192.008 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:57:21:192.037 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:57:21:192.075 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c104c0 2025-03-20 Thu 01:57:21:192.102 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0e900 2025-03-20 Thu 01:57:21:192.115 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0dfc0 2025-03-20 Thu 01:57:21:192.137 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0fb80 2025-03-20 Thu 01:57:21:192.151 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0f240 2025-03-20 Thu 01:57:21:192.334 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0bac0 2025-03-20 Thu 01:57:21:192.442 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0cd40 2025-03-20 Thu 01:57:21:192.464 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0b180 2025-03-20 Thu 01:57:21:192.523 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0d680 2025-03-20 Thu 01:57:21:207.576 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:57:21:208.437 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: -1 0x2c0c400 2025-03-20 Thu 01:57:26:018.632 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a840 2025-03-20 Thu 01:57:26:018.779 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0a0d0 2025-03-20 Thu 01:57:26:018.823 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c03070 2025-03-20 Thu 01:57:26:018.872 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09cd0 2025-03-20 Thu 01:57:26:019.061 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c09620 2025-03-20 Thu 01:57:26:019.134 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0a840 2025-03-20 Thu 01:57:26:019.143 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:019.165 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:019.177 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0a0d0 2025-03-20 Thu 01:57:26:019.183 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:019.192 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:019.201 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c03070 2025-03-20 Thu 01:57:26:019.214 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:019.223 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:019.231 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c09cd0 2025-03-20 Thu 01:57:26:019.236 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:019.244 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:019.261 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c09620 2025-03-20 Thu 01:57:26:019.266 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:019.274 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:020.176 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:020.198 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:020.221 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:020.231 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:020.244 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:020.252 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:020.264 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:020.273 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:020.285 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:020.293 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:020.317 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:020.846 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:26:020.879 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: SANDESH: Send FAILED: 1742435846020345 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:26:020.894 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082728113920, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:26:020.912 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:020.931 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:020.946 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:020.961 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:192.132 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c104c0 2025-03-20 Thu 01:57:26:192.280 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0e900 2025-03-20 Thu 01:57:26:192.315 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0dfc0 2025-03-20 Thu 01:57:26:192.360 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0fb80 2025-03-20 Thu 01:57:26:192.391 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0f240 2025-03-20 Thu 01:57:26:192.425 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0bac0 2025-03-20 Thu 01:57:26:192.481 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0cd40 2025-03-20 Thu 01:57:26:192.523 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0b180 2025-03-20 Thu 01:57:26:192.592 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c104c0 2025-03-20 Thu 01:57:26:192.606 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:192.633 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:192.646 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0d680 2025-03-20 Thu 01:57:26:192.720 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0e900 2025-03-20 Thu 01:57:26:192.729 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:192.740 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:192.768 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0dfc0 2025-03-20 Thu 01:57:26:192.784 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:192.796 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:192.804 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0fb80 2025-03-20 Thu 01:57:26:192.810 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:192.820 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:192.840 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0f240 2025-03-20 Thu 01:57:26:192.846 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:192.854 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:192.874 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0bac0 2025-03-20 Thu 01:57:26:192.882 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:192.892 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:192.915 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0cd40 2025-03-20 Thu 01:57:26:192.921 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:192.931 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:192.952 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0b180 2025-03-20 Thu 01:57:26:192.957 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:192.965 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:192.990 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:193.003 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:193.032 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:193.049 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:193.058 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:193.537 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:26:193.568 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SANDESH: Send FAILED: 1742435846193093 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:26:193.577 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:26:193.615 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:193.633 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:193.643 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:193.662 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:193.675 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:193.684 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:193.692 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0d680 2025-03-20 Thu 01:57:26:193.698 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:193.707 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:193.720 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:193.729 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:193.741 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:193.750 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:193.767 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:193.776 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:193.788 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:193.797 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:193.814 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:193.832 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:193.851 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:196.427 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083189118720, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:26:196.465 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083189118720, Pid 1]: SANDESH: Send FAILED: 1742435846193870 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:26:196.473 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083189118720, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:26:196.492 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:196.510 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:197.009 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:197.023 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:197.073 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:209.975 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x2c0c400 2025-03-20 Thu 01:57:26:210.917 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: RAC_Connect status: 0 0x2c0c400 2025-03-20 Thu 01:57:26:210.938 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:210.961 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:211.679 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:211.702 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:211.723 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:29:663.162 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:57:29:663.351 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-03-20 Thu 01:57:29:663.465 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:57:29:663.493 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SANDESH: Send FAILED: 1742435849663238 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:29:663.502 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:29:663.511 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:57:29:663.650 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SANDESH: Send FAILED: 1742435849663446 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 41 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 54 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 247, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 96, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:57:29:663.671 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:29:666.223 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 01:57:29:666.243 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 01:57:29:668.880 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 01:57:29:668.893 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 01:57:29:669.013 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:57:29:669.043 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Idle 2025-03-20 Thu 01:57:29:669.100 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:29:669.246 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435849668963 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 20 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 247, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 98, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:57:29:669.267 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:29:669.278 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:29:669.299 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435849668988 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:29:669.306 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:29:669.315 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:57:29:669.324 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:29:669.449 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435849669091 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 55 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 248, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 98, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:57:29:669.463 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:40:838.418 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:40:838.517 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SANDESH: Send FAILED: 1742435860838299 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 463 dequeues = 463 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 515 dequeues = 515 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742435849669042 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742435849669322 ] max_sm_queue_count = 6 ] 2025-03-20 Thu 01:57:40:838.535 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:40:838.549 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:40:838.715 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SANDESH: Send FAILED: 1742435860838391 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 57 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 248, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 101, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:57:40:838.753 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:45:824.458 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:45:824.543 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435865824426 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-03-20 Thu 01:57:45:824.554 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-03-20 Thu 01:57:59:685.616 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:57:59:685.786 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Connect : Start Connect timer 10.0.0.47:8086 2025-03-20 Thu 01:57:59:685.876 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:57:59:685.925 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SANDESH: Send FAILED: 1742435879685684 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:59:685.934 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:59:685.943 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:57:59:686.096 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: SANDESH: Send FAILED: 1742435879685865 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 8 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 59 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 21 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 248, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 104, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:57:59:686.111 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082723915520, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:59:710.015 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:34210::10.0.0.47:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:57:59:710.054 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:34210::10.0.0.47:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:57:59:710.062 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083362179200, Pid 1]: 10.0.0.42:34210::10.0.0.47:8086(-1):EvTcpConnected 2025-03-20 Thu 01:57:59:712.741 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:57:59:712.756 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:57:59:712.792 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: ClientInit 2025-03-20 Thu 01:57:59:712.804 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-1.:contrail-query-engine:0:Database count 9 2025-03-20 Thu 01:57:59:712.888 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:57:59:712.897 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:57:59:712.922 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435879712785 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:59:712.929 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:57:59:712.938 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:57:59:712.945 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:57:59:712.956 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:57:59:712.962 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:57:59:713.110 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SANDESH: Send FAILED: 1742435879712878 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 9 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 60 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 249, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 106, _iter59->first = sent _iter59->second = 361, ] ] ] 2025-03-20 Thu 01:57:59:713.131 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:57:59:720.095 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: OnMessage control in state: ClientInit 2025-03-20 Thu 01:57:59:720.148 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-03-20 Thu 01:57:59:720.158 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-03-20 Thu 01:57:59:720.182 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Received Ctrl Message with size 0 2025-03-20 Thu 01:57:59:720.404 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 9 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 4481 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3336423 calls = 354 average_bytes = 9424 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 463 dequeues = 463 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 515 dequeues = 515 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742435849669042 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742435849669322 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 21 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 176500 calls = 18 average_bytes = 9805 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 14 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 44 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 69 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 8 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 270 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 60 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 249, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 106, _iter59->first = sent _iter59->second = 361, ] ] max_sm_queue_count = 6 proxy seq 329 2025-03-20 Thu 01:57:59:720.436 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:57:59:720.462 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 57 2025-03-20 Thu 01:57:59:720.478 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:57:59:720.532 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.4 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 69 2025-03-20 Thu 01:57:59:720.543 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:57:59:720.556 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 01:57:59:720.587 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Established 2025-03-20 Thu 01:58:10:838.896 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 69 without seqno , total = 0 2025-03-20 Thu 01:59:10:844.901 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083184252672, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 70 without seqno , total = 0 2025-03-20 Thu 02:00:10:849.586 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083214309120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 71 without seqno , total = 0 2025-03-20 Thu 02:01:10:853.400 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140082719717120, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 72 without seqno , total = 0 2025-03-20 Thu 02:01:23:747.573 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvStop in state Established 2025-03-20 Thu 02:01:23:747.671 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Idle 2025-03-20 Thu 02:01:23:747.771 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 02:01:23:747.782 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 02:01:23:747.923 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: SANDESH: Send FAILED: 1742436083747760 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 9 pid = 1 http_port = 8091 start_time = 1742431715428975 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 5498 calls = 10 average_bytes = 549 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3558901 calls = 375 average_bytes = 9490 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 17 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 73 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 226 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 9 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 286 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 61 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 22 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 249, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 108, _iter59->first = sent _iter59->second = 385, ] ] ] 2025-03-20 Thu 02:01:23:747.947 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140083218507520, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 02:01:52:172.601 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: http-server-port 8091 2025-03-20 Thu 02:01:52:172.683 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Max-tasks 16 2025-03-20 Thu 02:01:52:172.692 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Max-slice 100 2025-03-20 Thu 02:01:52:172.698 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Collectors 10.0.0.42:8086 2025-03-20 Thu 02:01:52:172.703 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Collectors 10.0.0.12:8086 2025-03-20 Thu 02:01:52:172.708 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Collectors 10.0.0.47:8086 2025-03-20 Thu 02:01:52:172.730 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: ROLE : Generator 2025-03-20 Thu 02:01:52:172.737 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-03-20 Thu 02:01:52:172.742 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2923-1. 2025-03-20 Thu 02:01:52:172.747 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: NODE TYPE : Database 2025-03-20 Thu 02:01:52:172.753 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: INSTANCE ID : 0 2025-03-20 Thu 02:01:52:172.758 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-03-20 Thu 02:01:52:172.764 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-03-20 Thu 02:01:52:173.367 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: HTTP Introspect Init 2025-03-20 Thu 02:01:52:173.472 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-03-20 Thu 02:01:52:173.538 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Sandesh Http Server Port: 8091 2025-03-20 Thu 02:01:52:173.578 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-03-20 Thu 02:01:52:173.587 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-03-20 Thu 02:01:52:173.662 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Idle 2025-03-20 Thu 02:01:52:173.825 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: No Client: 1742436112173755 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742436112173677 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 02:01:52:174.065 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-03-20 Thu 02:01:52:174.093 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-03-20 Thu 02:01:52:174.109 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Cassandra Servers: 10.0.0.42:9042 10.0.0.12:9042 10.0.0.47:9042 2025-03-20 Thu 02:01:52:177.567 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Processing scm::EvStart in state Idle 2025-03-20 Thu 02:01:52:177.720 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Disconnect 2025-03-20 Thu 02:01:52:177.791 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-03-20 Thu 02:01:52:178.027 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Connect : Start Connect timer 10.0.0.47:8086 2025-03-20 Thu 02:01:52:178.101 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:01:52:178.122 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: SANDESH: Send FAILED: 1742436112177711 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-03-20 Thu 02:01:52:178.136 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 02:01:52:178.145 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:01:52:178.236 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: SANDESH: Send FAILED: 1742436112177781 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742436112173677 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 02:01:52:178.248 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 02:01:52:178.257 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:01:52:178.349 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: SANDESH: Send FAILED: 1742436112177837 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742436112173677 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 02:01:52:178.363 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 02:01:52:178.373 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:01:52:178.388 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: SANDESH: Send FAILED: 1742436112177855 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-03-20 Thu 02:01:52:178.395 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 02:01:52:178.402 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:01:52:178.464 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: SANDESH: Send FAILED: 1742436112178092 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742436112173677 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 02:01:52:178.476 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 02:01:52:213.530 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-03-20 Thu 02:01:52:217.067 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1437470 2025-03-20 Thu 02:01:52:217.104 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.123 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.136 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1437ca0 2025-03-20 Thu 02:01:52:217.142 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.159 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.168 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1438520 2025-03-20 Thu 02:01:52:217.173 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.181 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.190 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1438da0 2025-03-20 Thu 02:01:52:217.205 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.213 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.223 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1474760 2025-03-20 Thu 02:01:52:217.228 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.235 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.244 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1474fc0 2025-03-20 Thu 02:01:52:217.249 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.257 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.266 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1477a40 2025-03-20 Thu 02:01:52:217.272 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.279 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.289 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x14782c0 2025-03-20 Thu 02:01:52:217.294 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.302 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.310 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x14760c0 2025-03-20 Thu 02:01:52:217.315 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.323 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.331 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1476940 2025-03-20 Thu 02:01:52:217.340 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.347 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.355 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1475840 2025-03-20 Thu 02:01:52:217.361 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.368 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.377 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1479490 2025-03-20 Thu 02:01:52:217.382 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.390 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.398 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1478b40 2025-03-20 Thu 02:01:52:217.403 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.410 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.419 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x1479de0 2025-03-20 Thu 02:01:52:217.424 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.431 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:217.440 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: RAC_Connect status: 0 0x14771c0 2025-03-20 Thu 02:01:52:217.445 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:01:52:217.452 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:01:52:218.283 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:01:52:218.330 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: SANDESH: Send FAILED: 1742436112216967 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-03-20 Thu 02:01:52:218.341 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 02:01:52:220.267 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:58216::10.0.0.47:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-03-20 Thu 02:01:52:220.288 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: 10.0.0.42:58216::10.0.0.47:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 02:01:52:220.296 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: 10.0.0.42:58216::10.0.0.47:8086(-1):EvTcpConnected 2025-03-20 Thu 02:01:52:220.562 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:220.583 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:220.645 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:220.664 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:220.673 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:220.686 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:220.695 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:220.707 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:220.716 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:220.728 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:220.737 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:220.762 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:220.779 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:220.794 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:220.808 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:220.875 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 02:01:52:220.889 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 02:01:52:220.912 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: ClientInit 2025-03-20 Thu 02:01:52:220.934 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-1.:contrail-query-engine:0:Database count 1 2025-03-20 Thu 02:01:52:221.017 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:01:52:221.027 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 02:01:52:221.056 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: SANDESH: Send FAILED: 1742436112220636 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 02:01:52:221.067 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 02:01:52:221.080 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:01:52:221.090 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 02:01:52:221.111 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: SANDESH: Send FAILED: 1742436112220756 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 02:01:52:221.118 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 02:01:52:221.126 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:01:52:221.132 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 02:01:52:221.155 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: SANDESH: Send FAILED: 1742436112220907 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 02:01:52:221.166 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 02:01:52:221.173 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:01:52:221.180 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 02:01:52:221.211 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:01:52:221.218 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 02:01:52:221.328 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: SANDESH: Send FAILED: 1742436112221008 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742436112173677 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 31 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 33, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 02:01:52:221.341 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 02:01:52:222.147 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:222.167 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:222.181 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:222.189 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:222.209 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:222.218 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:222.235 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:222.243 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:222.256 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:222.264 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:222.277 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:222.286 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:222.299 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:222.307 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:222.319 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:222.328 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:222.340 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:222.349 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:222.361 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:01:52:222.370 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:01:52:222.393 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:222.431 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:222.447 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:222.464 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:222.477 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:222.497 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:222.528 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:222.550 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:222.565 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:222.577 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:01:52:222.622 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:01:52:222.631 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 02:01:52:222.652 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: SANDESH: Send FAILED: 1742436112222413 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 02:01:52:222.660 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 02:01:52:222.668 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:01:52:222.675 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 02:01:52:222.692 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: SANDESH: Send FAILED: 1742436112222519 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 02:01:52:222.702 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526981592832, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 02:01:52:223.726 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: OnMessage control in state: ClientInit 2025-03-20 Thu 02:01:52:223.829 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-03-20 Thu 02:01:52:223.841 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-03-20 Thu 02:01:52:223.867 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: Received Ctrl Message with size 0 2025-03-20 Thu 02:01:52:224.007 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742436112173677 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 31 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 33, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-03-20 Thu 02:01:52:224.039 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 02:01:52:224.067 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-1. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-03-20 Thu 02:01:52:224.080 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 02:01:52:224.088 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-03-20 Thu 02:01:52:224.095 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 02:01:52:224.127 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: Established 2025-03-20 Thu 02:02:52:174.380 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526720349952, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-03-20 Thu 02:03:52:175.885 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526720349952, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-03-20 Thu 02:04:52:176.390 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526720349952, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-03-20 Thu 02:05:52:177.627 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526737143552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-03-20 Thu 02:06:52:182.386 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-03-20 Thu 02:07:52:182.938 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-03-20 Thu 02:08:52:183.401 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-03-20 Thu 02:09:52:186.783 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-03-20 Thu 02:10:52:187.255 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526737143552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-03-20 Thu 02:11:52:187.756 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526737143552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-03-20 Thu 02:12:52:188.296 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-03-20 Thu 02:13:52:188.835 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526737143552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-03-20 Thu 02:14:52:191.075 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526989989632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-03-20 Thu 02:15:52:191.550 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526737143552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-03-20 Thu 02:16:52:192.064 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526985791232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-03-20 Thu 02:17:52:192.608 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526998386432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-03-20 Thu 02:18:19:048.323 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:8091::10.0.0.42:46254 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-03-20 Thu 02:18:37:776.671 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:8091::10.0.0.42:44716 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-03-20 Thu 02:18:52:193.207 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527002584832, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-03-20 Thu 02:19:52:195.213 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-03-20 Thu 02:20:01:464.508 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:8091::10.0.0.42:60178 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-03-20 Thu 02:20:23:385.441 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.42 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-03-20 Thu 02:20:23:385.699 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID e0efbd80-0531-11f0-82dc-00000a00002a src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-03-20 Thu 02:20:23:385.746 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a :0:json query is: { end_time: 1742437223321463 , enqueue_time: 1742437223385095 , query_metadata: {"enqueue_time": 1742437223384988} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:23:385.947 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-03-20 Thu 02:20:23:385.965 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-03-20 Thu 02:20:23:386.013 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for e0efbd80-0531-11f0-82dc-00000a00002a , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-03-20 Thu 02:20:23:386.100 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a :0:json query is: { end_time: 1742437223321463 , enqueue_time: 1742437223385095 , query_metadata: {"enqueue_time": 1742437223384988} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:23:536.857 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:536.900 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:536.911 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:536.933 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:536.941 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 0 and 33 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:537.004 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 33 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:537.018 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 33 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:537.039 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 33 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:537.057 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 33 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:537.074 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 33 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:537.180 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a :0:json query is: { end_time: 1742437223321463 , enqueue_time: 1742437223385095 , query_metadata: {"enqueue_time": 1742437223384988} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectVNTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:23:537.434 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:537.582 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:537.708 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:537.839 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:537.977 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:538.120 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:538.241 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:538.362 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:538.482 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:538.604 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:538.725 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:538.850 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:538.970 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:539.095 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:539.225 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:539.521 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 0 and 11 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:539.554 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 11 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:539.566 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 11 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:539.578 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 11 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:539.587 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 11 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:539.598 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 11 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:539.607 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 11 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:539.617 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 11 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:539.626 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 11 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:539.636 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 11 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:539.836 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:539.964 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:540.102 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:540.224 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:540.344 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:540.464 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:540.585 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:540.704 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:540.824 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:541.096 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:541.225 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :1:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:541.621 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:541.746 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:541.876 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:542.008 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:542.134 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:542.254 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:542.374 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:542.493 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:542.613 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:542.734 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:542.853 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:542.972 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:543.098 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:543.223 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:543.342 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:543.461 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:543.579 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:543.696 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QEQueryLog: e0efbd80-0531-11f0-82dc-00000a00002a ObjectVNTable :0:Column ObjectTypeName1 Table ObjectVNTable object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:23:543.919 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:543.933 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:543.941 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:543.948 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:543.956 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:543.963 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:543.971 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:543.978 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:543.989 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:23:543.996 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526724548352, Pid 1]: [SYS_INFO]: QELog: e0efbd80-0531-11f0-82dc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:23:544.253 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-03-20 Thu 02:20:23:544.879 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 44 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-03-20 Thu 02:20:23:544.992 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":6, "count":44} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-03-20 Thu 02:20:23:546.562 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: Finished: QID e0efbd80-0531-11f0-82dc-00000a00002a Table ObjectVNTable Time(ms) 160 RedisTime(ms) 0 MergeTime(ms) 0 Rows 44 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-03-20 Thu 02:20:23:546.663 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-03-20 Thu 02:20:23:546.708 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2923-1. table = ObjectVNTable query_stats= [ qid = e0efbd80-0531-11f0-82dc-00000a00002a where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "Messagetype", "ModuleId", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 150,0, 156,0, 153,0, chunk_select_time = 0,6, 0,0, 0,1, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 160 rows = 44 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-03-20 Thu 02:20:26:516.056 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.42 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-03-20 Thu 02:20:26:516.458 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID e2cd53e6-0531-11f0-a7bc-00000a00002a src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-03-20 Thu 02:20:26:516.511 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QEQueryLog: e2cd53e6-0531-11f0-a7bc-00000a00002a :0:json query is: { end_time: 1742437226495609 , enqueue_time: 1742437226515603 , query_metadata: {"enqueue_time": 1742437226515503} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "87d6e16b-1ec1-45c8-b000-792372cbfae7", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:26:516.712 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-03-20 Thu 02:20:26:516.738 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-03-20 Thu 02:20:26:516.810 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for e2cd53e6-0531-11f0-a7bc-00000a00002a , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-03-20 Thu 02:20:26:517.625 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526728746752, Pid 1]: [SYS_INFO]: QEQueryLog: e2cd53e6-0531-11f0-a7bc-00000a00002a :0:json query is: { end_time: 1742437226495609 , enqueue_time: 1742437226515603 , query_metadata: {"enqueue_time": 1742437226515503} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "87d6e16b-1ec1-45c8-b000-792372cbfae7", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:26:553.371 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:553.435 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:553.446 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:553.455 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:553.463 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:553.471 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:553.497 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:553.509 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:553.521 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:553.533 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:554.987 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:555.019 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:555.029 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:555.038 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:555.046 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:555.054 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:555.061 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 0 and 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:555.071 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:555.079 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:555.100 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:555.333 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QEQueryLog: e2cd53e6-0531-11f0-a7bc-00000a00002a ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 87d6e16b-1ec1-45c8-b000-792372cbfae7 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:26:555.477 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: [SYS_INFO]: QEQueryLog: e2cd53e6-0531-11f0-a7bc-00000a00002a ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 87d6e16b-1ec1-45c8-b000-792372cbfae7 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:26:558.970 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:559.005 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:559.014 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 0 and 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:559.027 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:559.036 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:559.045 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:559.053 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:559.062 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:559.069 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:26:559.077 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: e2cd53e6-0531-11f0-a7bc-00000a00002a Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:26:559.169 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QEQueryLog: e2cd53e6-0531-11f0-a7bc-00000a00002a :0:json query is: { end_time: 1742437226495609 , enqueue_time: 1742437226515603 , query_metadata: {"enqueue_time": 1742437226515503} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "87d6e16b-1ec1-45c8-b000-792372cbfae7", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:26:559.398 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QEQueryLog: e2cd53e6-0531-11f0-a7bc-00000a00002a ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id 87d6e16b-1ec1-45c8-b000-792372cbfae7 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:26:559.533 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QEQueryLog: e2cd53e6-0531-11f0-a7bc-00000a00002a ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id 87d6e16b-1ec1-45c8-b000-792372cbfae7 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:26:559.657 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QEQueryLog: e2cd53e6-0531-11f0-a7bc-00000a00002a ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id 87d6e16b-1ec1-45c8-b000-792372cbfae7 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:26:559.847 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-03-20 Thu 02:20:26:559.948 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-03-20 Thu 02:20:26:560.001 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526716151552, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-03-20 Thu 02:20:26:562.874 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: Finished: QID e2cd53e6-0531-11f0-a7bc-00000a00002a Table ObjectVMTable Time(ms) 46 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-03-20 Thu 02:20:26:562.960 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526766520064, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-03-20 Thu 02:20:26:563.020 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526973196032, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2923-1. table = ObjectVMTable query_stats= [ qid = e2cd53e6-0531-11f0-a7bc-00000a00002a where = [[{"name": "ObjectId", "value": "87d6e16b-1ec1-45c8-b000-792372cbfae7", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 34,0, 37,0, 41,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 46 rows = 5 enq_delay = 0 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-03-20 Thu 02:20:52:196.978 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527006783232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-03-20 Thu 02:21:52:197.546 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526998386432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-03-20 Thu 02:22:52:198.079 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526985791232, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-03-20 Thu 02:23:52:198.640 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526977394432, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-03-20 Thu 02:24:52:199.558 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-03-20 Thu 02:25:52:200.051 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-03-20 Thu 02:26:52:201.725 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526732945152, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-03-20 Thu 02:27:14:472.321 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:8091::10.0.0.42:37972 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-03-20 Thu 02:27:52:202.293 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526989989632, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-03-20 Thu 02:28:52:204.859 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140526994188032, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-03-20 Thu 02:29:13:466.859 UTC cn-jenkins-deploy-platform-ansible-os-2923-1 [Thread 140527154645120, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.42:8091::10.0.0.42:32796 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256