2025-03-20 Thu 00:48:35:131.844 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: http-server-port 8091 2025-03-20 Thu 00:48:35:131.931 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Max-tasks 16 2025-03-20 Thu 00:48:35:131.939 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Max-slice 100 2025-03-20 Thu 00:48:35:131.944 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Collectors 10.0.0.42:8086 2025-03-20 Thu 00:48:35:131.949 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Collectors 10.0.0.12:8086 2025-03-20 Thu 00:48:35:131.953 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Collectors 10.0.0.47:8086 2025-03-20 Thu 00:48:35:131.971 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: ROLE : Generator 2025-03-20 Thu 00:48:35:131.978 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-03-20 Thu 00:48:35:131.982 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2923-2. 2025-03-20 Thu 00:48:35:131.987 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: NODE TYPE : Database 2025-03-20 Thu 00:48:35:131.992 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: INSTANCE ID : 0 2025-03-20 Thu 00:48:35:131.996 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-03-20 Thu 00:48:35:132.001 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-03-20 Thu 00:48:35:132.535 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: HTTP Introspect Init 2025-03-20 Thu 00:48:35:132.647 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-03-20 Thu 00:48:35:132.702 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Sandesh Http Server Port: 8091 2025-03-20 Thu 00:48:35:132.739 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-03-20 Thu 00:48:35:132.747 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-03-20 Thu 00:48:35:132.818 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Idle 2025-03-20 Thu 00:48:35:132.962 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: No Client: 1742431715132893 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715132827 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:48:35:133.097 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-03-20 Thu 00:48:35:133.110 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-03-20 Thu 00:48:35:133.121 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Cassandra Servers: 10.0.0.42:9042 10.0.0.12:9042 10.0.0.47:9042 2025-03-20 Thu 00:48:35:133.199 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Processing scm::EvStart in state Idle 2025-03-20 Thu 00:48:35:133.270 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Disconnect 2025-03-20 Thu 00:48:35:133.339 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-03-20 Thu 00:48:35:133.536 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Connect : Start Connect timer 10.0.0.42:8086 2025-03-20 Thu 00:48:35:133.611 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:133.631 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: SANDESH: Send FAILED: 1742431715133262 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-03-20 Thu 00:48:35:133.639 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:48:35:133.647 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:133.755 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: SANDESH: Send FAILED: 1742431715133329 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715132827 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:48:35:133.777 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:48:35:133.790 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:133.869 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: SANDESH: Send FAILED: 1742431715133386 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715132827 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:48:35:133.882 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:48:35:133.891 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:133.905 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: SANDESH: Send FAILED: 1742431715133406 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-03-20 Thu 00:48:35:133.911 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:48:35:133.919 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:133.980 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: SANDESH: Send FAILED: 1742431715133600 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715132827 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:48:35:133.991 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:48:35:139.617 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-03-20 Thu 00:48:35:139.661 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:48:35:139.733 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 0 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:48:35:139.775 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282350679808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:48:35:139.814 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282350679808, Pid 1]: SANDESH: Send FAILED: 1742431715139705 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-03-20 Thu 00:48:35:139.825 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282350679808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:48:40:146.789 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-03-20 Thu 00:48:40:146.835 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:48:40:146.876 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 1 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:48:45:150.974 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-03-20 Thu 00:48:45:151.019 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:48:45:151.040 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 2 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:48:50:153.204 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Database initialization failed src/contrail-analytics/contrail-query-engine/query.cc 997 2025-03-20 Thu 00:48:50:153.248 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:48:50:153.275 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 3 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:48:55:303.507 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:48:55:303.567 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:48:55:303.588 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 4 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:00:331.396 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:00:331.443 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:00:331.464 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 5 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:05:353.004 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:05:353.052 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:05:353.072 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 6 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:10:378.819 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:10:378.872 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:10:378.893 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 7 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:15:401.355 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:15:401.404 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:15:401.425 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 8 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:20:421.423 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:20:421.477 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:20:421.507 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 9 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:25:439.917 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: : Create/Set KEYSPACE: ContrailAnalyticsCql FAILED src/contrail-analytics/contrail-query-engine/query.cc 1004 2025-03-20 Thu 00:49:25:485.241 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:25:485.284 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 10 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:30:505.695 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table exists. will query both the tables src/contrail-analytics/contrail-query-engine/query.cc 1043 2025-03-20 Thu 00:49:30:505.751 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: QeInit [SYS_WARN]: QELog: initialization of database failed. retrying 11 time src/contrail-analytics/contrail-query-engine/query.cc 1065 2025-03-20 Thu 00:49:35:530.145 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-03-20 Thu 00:49:40:537.583 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:49:40:537.644 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 00:49:40:537.679 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 00:49:40:537.697 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c961a0 2025-03-20 Thu 00:49:40:537.713 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.726 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.737 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c96ae0 2025-03-20 Thu 00:49:40:537.742 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.750 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.757 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c97420 2025-03-20 Thu 00:49:40:537.762 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.770 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.778 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c97d60 2025-03-20 Thu 00:49:40:537.782 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.789 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.798 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c945f0 2025-03-20 Thu 00:49:40:537.802 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.810 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.817 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c986a0 2025-03-20 Thu 00:49:40:537.821 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.829 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.836 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c8e1d0 2025-03-20 Thu 00:49:40:537.840 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.848 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.855 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c94ca0 2025-03-20 Thu 00:49:40:537.860 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.867 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.874 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c95340 2025-03-20 Thu 00:49:40:537.881 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.888 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.896 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c95870 2025-03-20 Thu 00:49:40:537.901 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.908 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.915 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c98fe0 2025-03-20 Thu 00:49:40:537.920 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.927 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.934 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c99920 2025-03-20 Thu 00:49:40:537.938 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.945 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.952 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c9a260 2025-03-20 Thu 00:49:40:537.957 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.966 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.973 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c9aba0 2025-03-20 Thu 00:49:40:537.978 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:537.985 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:537.992 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: RAC_Connect status: 0 0x1c9b4e0 2025-03-20 Thu 00:49:40:537.997 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: Connected to REDIS... 2025-03-20 Thu 00:49:40:538.004 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 00:49:40:538.411 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:537.654 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SANDESH: Send FAILED: 1742431780537488 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-03-20 Thu 00:49:40:538.443 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:538.491 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:538.505 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:538.469 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:40:538.524 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:538.540 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 00:49:40:538.562 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:538.583 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:538.567 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 00:49:40:538.598 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:538.622 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:538.633 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:538.691 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:538.735 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:538.735 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 00:49:40:538.759 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:538.783 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:538.788 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Idle 2025-03-20 Thu 00:49:40:538.800 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:538.840 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:538.851 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:538.870 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:538.884 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:538.893 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:539.002 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:539.031 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:539.038 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:539.060 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:539.063 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SANDESH: Send FAILED: 1742431780538678 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 00:49:40:539.082 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:40:539.091 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:539.095 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:539.112 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:539.125 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:539.139 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:539.162 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:539.180 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:539.199 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:539.215 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SANDESH: Send FAILED: 1742431780538683 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715132827 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 62, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:49:40:539.266 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:49:40:539.276 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:539.218 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:539.300 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SANDESH: Send FAILED: 1742431780538708 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 00:49:40:539.311 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:40:539.301 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:539.320 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:539.347 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:539.354 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SANDESH: Send FAILED: 1742431780538727 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.42:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 00:49:40:539.362 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:539.365 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:40:539.389 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 00:49:40:539.427 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:539.380 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:539.445 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SANDESH: Send FAILED: 1742431780538914 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.42:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 00:49:40:539.453 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:40:539.466 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:539.458 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:539.496 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:539.518 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:539.529 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:539.600 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:539.601 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SANDESH: Send FAILED: 1742431780539019 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715132827 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 72 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 74, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 7, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:49:40:539.627 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:539.637 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:49:40:539.658 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:539.661 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:539.691 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SANDESH: Send FAILED: 1742431780539591 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 00:49:40:539.697 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:40:539.776 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:539.817 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SANDESH: Send FAILED: 1742431780539716 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742431780538787 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742431780539659 ] max_sm_queue_count = 7 ] 2025-03-20 Thu 00:49:40:539.824 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:49:40:539.834 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 00:49:40:539.875 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 00:49:40:539.889 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 00:49:40:539.913 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 00:49:40:539.941 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SANDESH: Send FAILED: 1742431780539764 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715132827 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 93 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 5 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 95, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 14, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:49:40:539.953 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:49:41:539.056 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 00:49:41:539.301 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-03-20 Thu 00:49:41:539.413 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:49:41:539.444 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: SANDESH: Send FAILED: 1742431781539134 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 00:49:41:539.452 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 00:49:41:539.461 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 00:49:41:539.581 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: SANDESH: Send FAILED: 1742431781539399 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742431715132827 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 98, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:49:41:539.603 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 00:49:41:541.223 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:50606::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 00:49:41:541.345 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: 10.0.0.12:50606::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 00:49:41:541.354 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: 10.0.0.12:50606::10.0.0.12:8086(-1):EvTcpConnected 2025-03-20 Thu 00:49:41:541.402 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 00:49:41:541.412 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 00:49:41:541.441 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: ClientInit 2025-03-20 Thu 00:49:41:541.456 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-2.:contrail-query-engine:0:Database count 1 2025-03-20 Thu 00:49:41:541.529 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 00:49:41:541.538 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 00:49:41:541.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: SANDESH: Send FAILED: 1742431781541435 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 00:49:41:541.577 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 00:49:41:541.586 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 00:49:41:541.592 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 00:49:41:541.602 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 00:49:41:541.607 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 00:49:41:541.713 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: SANDESH: Send FAILED: 1742431781541520 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742431715132827 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 00:49:41:541.731 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 00:49:41:542.756 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: OnMessage control in state: ClientInit 2025-03-20 Thu 00:49:41:542.792 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-03-20 Thu 00:49:41:542.802 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-03-20 Thu 00:49:41:542.824 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: Received Ctrl Message with size 0 2025-03-20 Thu 00:49:41:542.983 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742431715132827 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 14 dequeues = 14 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 18 dequeues = 18 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742431780538787 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742431780539659 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 99, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 0, ] ] max_sm_queue_count = 7 proxy seq 10 2025-03-20 Thu 00:49:41:543.008 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 00:49:41:543.031 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 11 2025-03-20 Thu 00:49:41:543.042 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 00:49:41:543.049 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-03-20 Thu 00:49:41:543.058 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 00:49:41:543.084 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: Established 2025-03-20 Thu 00:49:50:713.699 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281726867200, Pid 1]: OnMessage control in state: Established 2025-03-20 Thu 00:49:50:715.007 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-03-20 Thu 00:49:50:715.039 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: Established : EvSandeshMessageRecv 2025-03-20 Thu 00:49:50:715.088 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: Received Ctrl Message with size 3 2025-03-20 Thu 00:49:50:715.100 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 11, total = 0 2025-03-20 Thu 00:49:50:715.110 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 13, total = 0 2025-03-20 Thu 00:49:50:715.121 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 1, total = 0 2025-03-20 Thu 00:49:50:715.128 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 00:50:10:541.969 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-03-20 Thu 00:51:10:542.661 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-03-20 Thu 00:52:10:544.328 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281726867200, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-03-20 Thu 00:52:16:157.223 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:8091::10.0.0.12:49584 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-03-20 Thu 00:52:59:139.380 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282498549888, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:8091::10.0.0.12:56494 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 254 2025-03-20 Thu 00:53:10:544.906 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281743660800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-03-20 Thu 00:54:10:545.591 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281743660800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-03-20 Thu 00:55:10:546.392 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-03-20 Thu 00:56:10:547.100 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281752057600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-03-20 Thu 00:57:10:548.009 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281726867200, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-03-20 Thu 00:58:10:548.580 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281735264000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-03-20 Thu 00:59:10:549.305 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281722668800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-03-20 Thu 01:00:10:549.970 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282295453440, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-03-20 Thu 01:01:10:550.479 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281722668800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-03-20 Thu 01:02:10:552.240 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281722668800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-03-20 Thu 01:03:10:553.326 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281731065600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-03-20 Thu 01:04:10:554.097 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-03-20 Thu 01:05:10:555.011 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282354878208, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-03-20 Thu 01:06:10:555.897 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281722668800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-03-20 Thu 01:07:10:556.602 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281752057600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-03-20 Thu 01:08:10:557.560 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281735264000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-03-20 Thu 01:09:10:558.438 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-03-20 Thu 01:10:10:559.799 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281722668800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-03-20 Thu 01:11:10:560.541 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281743660800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-03-20 Thu 01:12:10:561.176 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281726867200, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-03-20 Thu 01:13:10:561.832 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281722668800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-03-20 Thu 01:14:10:562.945 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-03-20 Thu 01:15:10:563.635 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282320631552, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 26 without seqno , total = 0 2025-03-20 Thu 01:16:10:564.182 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282329024256, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 27 without seqno , total = 0 2025-03-20 Thu 01:17:10:565.255 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281752057600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 28 without seqno , total = 0 2025-03-20 Thu 01:18:10:566.268 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281735264000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 29 without seqno , total = 0 2025-03-20 Thu 01:19:10:567.224 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 30 without seqno , total = 0 2025-03-20 Thu 01:20:10:571.622 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282329024256, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 31 without seqno , total = 0 2025-03-20 Thu 01:21:10:572.341 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 32 without seqno , total = 0 2025-03-20 Thu 01:22:10:572.993 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 33 without seqno , total = 0 2025-03-20 Thu 01:23:10:573.594 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281731065600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 34 without seqno , total = 0 2025-03-20 Thu 01:24:10:574.303 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 35 without seqno , total = 0 2025-03-20 Thu 01:25:10:574.885 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282329024256, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 36 without seqno , total = 0 2025-03-20 Thu 01:26:10:575.444 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281731065600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 37 without seqno , total = 0 2025-03-20 Thu 01:27:10:576.192 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 38 without seqno , total = 0 2025-03-20 Thu 01:28:10:576.887 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282350679808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 39 without seqno , total = 0 2025-03-20 Thu 01:29:10:577.488 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282350679808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 40 without seqno , total = 0 2025-03-20 Thu 01:30:10:578.107 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281731065600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 41 without seqno , total = 0 2025-03-20 Thu 01:31:10:579.287 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281731065600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 42 without seqno , total = 0 2025-03-20 Thu 01:32:10:580.128 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282350679808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 43 without seqno , total = 0 2025-03-20 Thu 01:33:10:580.690 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281743660800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 44 without seqno , total = 0 2025-03-20 Thu 01:34:10:581.320 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282346481408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 45 without seqno , total = 0 2025-03-20 Thu 01:35:10:581.970 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281726867200, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 46 without seqno , total = 0 2025-03-20 Thu 01:36:10:582.591 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282329024256, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 47 without seqno , total = 0 2025-03-20 Thu 01:37:10:583.325 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281743660800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 48 without seqno , total = 0 2025-03-20 Thu 01:38:10:584.463 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281747859200, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 49 without seqno , total = 0 2025-03-20 Thu 01:39:10:585.737 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 50 without seqno , total = 0 2025-03-20 Thu 01:40:10:586.468 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281747859200, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 51 without seqno , total = 0 2025-03-20 Thu 01:41:10:587.231 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281743660800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 52 without seqno , total = 0 2025-03-20 Thu 01:42:10:588.329 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281731065600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 53 without seqno , total = 0 2025-03-20 Thu 01:43:10:588.890 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282329024256, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 54 without seqno , total = 0 2025-03-20 Thu 01:44:10:589.545 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282350679808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 55 without seqno , total = 0 2025-03-20 Thu 01:45:10:590.169 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281722668800, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 56 without seqno , total = 0 2025-03-20 Thu 01:46:10:590.796 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 57 without seqno , total = 0 2025-03-20 Thu 01:47:10:591.540 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281718470400, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 58 without seqno , total = 0 2025-03-20 Thu 01:48:10:592.269 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281735264000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 59 without seqno , total = 0 2025-03-20 Thu 01:49:10:593.525 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281731065600, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 60 without seqno , total = 0 2025-03-20 Thu 01:50:10:595.141 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140281735264000, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 61 without seqno , total = 0 2025-03-20 Thu 01:51:10:595.775 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282329024256, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 62 without seqno , total = 0 2025-03-20 Thu 01:52:10:596.777 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 63 without seqno , total = 0 2025-03-20 Thu 01:52:36:780.479 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: Processing scm::EvStop in state Established 2025-03-20 Thu 01:52:36:780.574 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: Idle 2025-03-20 Thu 01:52:36:780.708 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:52:36:780.725 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:52:36:780.969 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: SANDESH: Send FAILED: 1742435556780689 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742431715132827 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, ] ] rx_socket_stats= [ bytes = 2447 calls = 4 average_bytes = 611 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 3041620 calls = 318 average_bytes = 9564 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 63 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 96 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 252 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 101, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 20, _iter59->first = sent _iter59->second = 319, ] ] ] 2025-03-20 Thu 01:52:36:781.004 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140282342283008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:52:52:283.388 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: http-server-port 8091 2025-03-20 Thu 01:52:52:283.527 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Max-tasks 16 2025-03-20 Thu 01:52:52:283.539 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Max-slice 100 2025-03-20 Thu 01:52:52:283.547 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Collectors 10.0.0.42:8086 2025-03-20 Thu 01:52:52:283.554 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Collectors 10.0.0.12:8086 2025-03-20 Thu 01:52:52:283.560 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Collectors 10.0.0.47:8086 2025-03-20 Thu 01:52:52:283.590 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: ROLE : Generator 2025-03-20 Thu 01:52:52:283.598 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-03-20 Thu 01:52:52:283.605 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2923-2. 2025-03-20 Thu 01:52:52:283.613 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: NODE TYPE : Database 2025-03-20 Thu 01:52:52:283.620 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: INSTANCE ID : 0 2025-03-20 Thu 01:52:52:283.627 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-03-20 Thu 01:52:52:283.636 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-03-20 Thu 01:52:52:284.490 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: HTTP Introspect Init 2025-03-20 Thu 01:52:52:284.640 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-03-20 Thu 01:52:52:284.722 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Sandesh Http Server Port: 8091 2025-03-20 Thu 01:52:52:284.756 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-03-20 Thu 01:52:52:284.764 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-03-20 Thu 01:52:52:284.843 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Idle 2025-03-20 Thu 01:52:52:285.081 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: No Client: 1742435572284961 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 01:52:52:285.222 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-03-20 Thu 01:52:52:285.238 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-03-20 Thu 01:52:52:285.261 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Cassandra Servers: 10.0.0.42:9042 10.0.0.12:9042 10.0.0.47:9042 2025-03-20 Thu 01:52:52:285.391 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvStart in state Idle 2025-03-20 Thu 01:52:52:285.482 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Disconnect 2025-03-20 Thu 01:52:52:285.560 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-03-20 Thu 01:52:52:285.768 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Connect : Start Connect timer 10.0.0.42:8086 2025-03-20 Thu 01:52:52:285.861 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:52:52:285.890 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435572285470 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-03-20 Thu 01:52:52:285.901 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:52:52:285.913 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:52:52:286.011 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435572285550 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 01:52:52:286.026 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:52:52:286.045 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:52:52:286.118 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435572285609 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 01:52:52:286.127 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:52:52:286.135 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:52:52:286.150 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435572285631 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-03-20 Thu 01:52:52:286.160 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:52:52:286.167 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:52:52:286.230 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435572285845 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 01:52:52:286.238 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:52:52:382.143 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-03-20 Thu 01:53:22:478.761 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398369060608, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:53:22:478.854 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398369060608, Pid 1]: SANDESH: Send FAILED: 1742435602478642 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-03-20 Thu 01:53:22:478.872 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398369060608, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:53:22:482.094 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:36674::10.0.0.42:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:53:22:482.133 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:36674::10.0.0.42:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:53:22:482.142 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:36674::10.0.0.42:8086(-1):EvTcpConnected 2025-03-20 Thu 01:53:22:482.197 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22de1a0 2025-03-20 Thu 01:53:22:482.203 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:22:482.220 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:53:22:482.222 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:22:482.288 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:53:22:482.300 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22deae0 2025-03-20 Thu 01:53:22:482.309 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:22:482.323 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:22:482.339 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22dc5f0 2025-03-20 Thu 01:53:22:482.347 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: ClientInit 2025-03-20 Thu 01:53:22:482.347 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:22:482.378 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-2.:contrail-query-engine:0:Database count 1 2025-03-20 Thu 01:53:22:482.378 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:22:482.424 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22dcca0 2025-03-20 Thu 01:53:22:482.433 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:22:482.460 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:22:482.474 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22df420 2025-03-20 Thu 01:53:22:482.482 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:22:482.495 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:22:482.510 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22d61d0 2025-03-20 Thu 01:53:22:482.523 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:22:482.512 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:53:22:482.538 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:22:482.562 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:53:22:482.585 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22dfd60 2025-03-20 Thu 01:53:22:482.606 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:22:482.613 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435602482337 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-03-20 Thu 01:53:22:482.618 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:22:482.628 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:53:22:482.646 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22dd340 2025-03-20 Thu 01:53:22:482.654 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:53:22:482.655 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:22:482.667 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:53:22:482.688 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:22:482.708 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:53:22:482.717 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22dd870 2025-03-20 Thu 01:53:22:482.747 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:22:482.736 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:53:22:482.763 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:22:482.794 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e06a0 2025-03-20 Thu 01:53:22:482.803 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:22:482.814 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:22:485.899 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:53:22:486.001 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:53:22:486.018 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:53:22:486.036 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:53:22:486.055 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:53:22:491.242 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:22:491.298 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:22:491.314 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:22:491.322 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:22:491.335 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:22:491.343 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:22:491.355 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:22:491.363 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:22:491.375 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:22:491.405 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:22:491.418 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:22:491.426 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:22:491.438 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:22:491.446 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:22:491.457 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:22:491.465 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:22:491.553 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:22:491.582 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:22:491.599 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:22:491.620 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:22:491.636 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:22:491.651 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:22:491.665 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:22:491.679 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:22:499.031 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:22:499.069 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:22:499.102 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:22:499.117 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:22:499.152 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:22:499.170 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:22:499.351 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435602482497 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 7, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 01:53:22:499.399 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:53:22:499.427 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:53:22:499.440 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:53:22:499.479 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435602491537 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:53:22:499.488 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:53:22:499.531 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:53:22:499.540 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:53:22:499.559 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435602491576 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.42:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:53:22:499.571 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:53:22:499.579 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:53:22:499.585 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:53:22:499.601 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435602491615 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:53:22:499.612 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:53:22:499.836 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:53:22:499.848 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:53:22:499.910 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435602499754 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1742435602482345 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1742435602499577 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 5 ] 2025-03-20 Thu 01:53:22:499.922 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:53:22:499.931 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:53:22:499.942 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:53:22:500.741 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742435602499818 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 11, _iter59->first = sent _iter59->second = 1, ] ] ] 2025-03-20 Thu 01:53:22:500.782 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:53:22:514.090 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: OnMessage control in state: ClientInit 2025-03-20 Thu 01:53:22:514.170 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-03-20 Thu 01:53:22:514.181 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-03-20 Thu 01:53:22:514.207 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Received Ctrl Message with size 0 2025-03-20 Thu 01:53:22:514.469 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 12 dequeues = 12 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 15 dequeues = 15 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1742435602482345 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1742435602499577 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 41 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 4 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 43, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 11, _iter59->first = sent _iter59->second = 1, ] ] max_sm_queue_count = 5 proxy seq 7 2025-03-20 Thu 01:53:22:514.514 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:53:22:514.543 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 7 2025-03-20 Thu 01:53:22:514.555 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:53:22:514.563 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-03-20 Thu 01:53:22:514.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 01:53:22:514.605 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Established 2025-03-20 Thu 01:53:27:486.007 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:53:27:486.154 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:53:27:486.206 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:53:27:486.243 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:53:27:486.293 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:53:27:486.643 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e0fe0 2025-03-20 Thu 01:53:27:486.670 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:27:486.700 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:27:486.765 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e34e0 2025-03-20 Thu 01:53:27:486.788 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:27:486.802 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:27:486.887 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e2260 2025-03-20 Thu 01:53:27:486.896 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:27:486.909 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:27:486.922 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e2ba0 2025-03-20 Thu 01:53:27:486.930 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:27:486.942 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:27:486.985 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e1920 2025-03-20 Thu 01:53:27:486.994 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:53:27:487.007 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:53:27:487.949 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:27:488.027 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:27:488.068 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:27:488.094 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:27:488.109 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:27:488.129 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:27:488.144 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:27:488.163 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:27:488.177 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:27:488.205 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:53:27:488.217 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:53:27:488.294 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:27:488.311 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:27:488.324 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:27:488.336 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:53:52:501.125 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-03-20 Thu 01:54:52:501.806 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398330619648, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-03-20 Thu 01:55:45:999.495 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22dd870 2025-03-20 Thu 01:55:45:999.558 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:45:999.691 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22d61d0 2025-03-20 Thu 01:55:45:999.706 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:45:999.732 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22dcca0 2025-03-20 Thu 01:55:45:999.742 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:45:999.762 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22dd340 2025-03-20 Thu 01:55:45:999.772 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:45:999.794 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-03-20 Thu 01:55:45:999.800 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22dc5f0 2025-03-20 Thu 01:55:45:999.810 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:000.004 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:55:46:001.397 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:55:46:001.426 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:55:46:001.440 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:55:46:001.454 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:55:46:005.568 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:36674::10.0.0.42:8086(-1) OnSessionEvent TCP Connection Closed 2025-03-20 Thu 01:55:46:005.599 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:36674::10.0.0.42:8086(-1):EvTcpClose 2025-03-20 Thu 01:55:46:005.682 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvTcpClose in state Established 2025-03-20 Thu 01:55:46:005.701 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Established : EvTcpClose 2025-03-20 Thu 01:55:46:006.066 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Connect : Start Connect timer 10.0.0.47:8086 2025-03-20 Thu 01:55:46:006.182 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:006.401 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435746005814 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = cn-jenkins-deploy-platform-ansible-os-2923-1. collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 144090 calls = 17 average_bytes = 8475 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 69, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 13, _iter59->first = sent _iter59->second = 19, ] ] ] 2025-03-20 Thu 01:55:46:006.436 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:006.452 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:006.485 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435746005855 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.42:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:55:46:006.496 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:46:006.508 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-03-20 Thu 01:55:46:006.567 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:007.011 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435746006165 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 144090 calls = 17 average_bytes = 8475 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 7 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 6 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 69, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 13, _iter59->first = sent _iter59->second = 19, ] ] ] 2025-03-20 Thu 01:55:46:007.050 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:012.644 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:35098::10.0.0.47:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:55:46:012.681 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:35098::10.0.0.47:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:55:46:012.691 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:35098::10.0.0.47:8086(-1):EvTcpConnected 2025-03-20 Thu 01:55:46:012.806 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:55:46:013.650 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:55:46:013.716 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: ClientInit 2025-03-20 Thu 01:55:46:013.747 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-2.:contrail-query-engine:0:Database count 2 2025-03-20 Thu 01:55:46:013.887 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:013.902 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:55:46:013.936 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: SANDESH: Send FAILED: 1742435746013703 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.42:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:55:46:013.947 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:55:46:013.959 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:013.969 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:55:46:013.985 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:013.994 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:014.217 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: SANDESH: Send FAILED: 1742435746013869 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 144090 calls = 17 average_bytes = 8475 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 70, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 19, ] ] ] 2025-03-20 Thu 01:55:46:014.265 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:049.417 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: OnMessage control in state: ClientInit 2025-03-20 Thu 01:55:46:050.462 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-03-20 Thu 01:55:46:050.479 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-03-20 Thu 01:55:46:050.506 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Received Ctrl Message with size 0 2025-03-20 Thu 01:55:46:050.806 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 2 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 144090 calls = 17 average_bytes = 8475 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 28 dequeues = 28 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 32 dequeues = 32 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Established state_since = 1742435602514604 last_state = Established last_event = scm::EvSandeshSend last_event_at = 1742435692501879 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 15 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 1017 calls = 2 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 116919 calls = 13 average_bytes = 8993 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 67 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 1 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 8 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 70, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 16, _iter59->first = sent _iter59->second = 19, ] ] max_sm_queue_count = 5 proxy seq 19 2025-03-20 Thu 01:55:46:050.864 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:55:46:050.903 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.42:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 13 2025-03-20 Thu 01:55:46:050.921 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:55:46:050.996 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 3 2025-03-20 Thu 01:55:46:051.014 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:55:46:051.025 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 01:55:46:051.074 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Established 2025-03-20 Thu 01:55:46:171.808 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22e06a0 2025-03-20 Thu 01:55:46:171.861 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:172.006 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22dfd60 2025-03-20 Thu 01:55:46:172.022 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:172.050 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22df420 2025-03-20 Thu 01:55:46:172.060 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:172.082 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22deae0 2025-03-20 Thu 01:55:46:172.091 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:172.115 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-03-20 Thu 01:55:46:172.121 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22de1a0 2025-03-20 Thu 01:55:46:172.129 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:172.352 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:55:46:172.373 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:55:46:172.385 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:55:46:172.404 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:55:46:172.415 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:55:46:172.430 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22e1920 2025-03-20 Thu 01:55:46:172.442 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..1 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:172.470 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22e2ba0 2025-03-20 Thu 01:55:46:172.481 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..3 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:172.500 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22e2260 2025-03-20 Thu 01:55:46:172.509 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..2 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:172.621 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: CallbackProcess: received NULL reply from redis src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1049 2025-03-20 Thu 01:55:46:172.630 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22e0fe0 2025-03-20 Thu 01:55:46:172.638 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..0 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:173.010 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Disconnect status: -1 0x22e34e0 2025-03-20 Thu 01:55:46:173.064 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnDown.. DOWN.. Reconnect..4 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1016 2025-03-20 Thu 01:55:46:173.108 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:55:46:173.491 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:55:46:173.521 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:55:46:173.538 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:55:46:173.656 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:55:46:192.806 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:35098::10.0.0.47:8086(-1) OnSessionEvent TCP Connection Closed 2025-03-20 Thu 01:55:46:192.845 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:35098::10.0.0.47:8086(-1):EvTcpClose 2025-03-20 Thu 01:55:46:192.932 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvTcpClose in state Established 2025-03-20 Thu 01:55:46:192.948 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Established : EvTcpClose 2025-03-20 Thu 01:55:46:193.385 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-03-20 Thu 01:55:46:193.606 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:193.779 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435746193060 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Established successful_connections = 2 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = cn-jenkins-deploy-platform-ansible-os-2923-3. collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 199280 calls = 23 average_bytes = 8664 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 82, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 28, ] ] ] 2025-03-20 Thu 01:55:46:193.798 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:193.828 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:193.862 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435746193103 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Established : EvTcpClose -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:46:193.877 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:46:193.895 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvTcpDeleteSession in state Connect 2025-03-20 Thu 01:55:46:193.934 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:46:194.135 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435746193583 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 2 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 199280 calls = 23 average_bytes = 8664 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 9 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 2 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 82, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 18, _iter59->first = sent _iter59->second = 28, ] ] ] 2025-03-20 Thu 01:55:46:194.194 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:202.385 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:49130::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:55:46:202.412 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:49130::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:55:46:202.420 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:49130::10.0.0.12:8086(-1):EvTcpConnected 2025-03-20 Thu 01:55:46:202.464 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:55:46:202.475 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:55:46:202.519 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: ClientInit 2025-03-20 Thu 01:55:46:202.539 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-2.:contrail-query-engine:0:Database count 3 2025-03-20 Thu 01:55:46:202.624 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:202.633 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:55:46:202.655 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435746202511 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:46:202.663 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:55:46:202.671 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:202.677 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:55:46:202.687 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:46:202.694 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:202.866 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435746202614 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 199280 calls = 23 average_bytes = 8664 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 10 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 2 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 21, _iter59->first = sent _iter59->second = 28, ] ] ] 2025-03-20 Thu 01:55:46:202.893 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:251.723 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:49130::10.0.0.12:8086(-1) OnSessionEvent TCP Connection Closed 2025-03-20 Thu 01:55:46:251.758 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:49130::10.0.0.12:8086(-1):EvTcpClose 2025-03-20 Thu 01:55:46:251.833 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-03-20 Thu 01:55:46:251.845 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: ClientInit : EvTcpClose 2025-03-20 Thu 01:55:46:252.017 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Return to idle with 5000 2025-03-20 Thu 01:55:46:252.034 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Idle 2025-03-20 Thu 01:55:46:252.133 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:46:252.432 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435746251955 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 3 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200853 calls = 24 average_bytes = 8368 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 29, ] ] ] 2025-03-20 Thu 01:55:46:252.468 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:46:252.482 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:46:252.511 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435746251998 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:46:252.518 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:46:252.527 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:55:46:252.554 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:46:252.755 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435746252118 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 3 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200853 calls = 24 average_bytes = 8368 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 11 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 23, _iter59->first = sent _iter59->second = 29, ] ] ] 2025-03-20 Thu 01:55:46:252.780 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:51:001.384 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:55:51:001.556 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:55:51:001.591 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:55:51:001.616 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:55:51:001.640 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:55:51:002.125 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:55:51:002.181 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:55:51:002.202 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:55:51:002.218 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:55:51:002.231 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:55:51:172.424 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:55:51:172.574 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:55:51:172.641 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:55:51:172.668 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:55:51:172.678 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:55:51:172.705 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:55:51:172.728 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:55:51:172.758 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:55:51:172.772 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:55:51:172.797 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:55:51:173.137 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:55:51:173.533 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:55:51:173.577 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:55:51:173.602 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:55:51:173.689 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:55:51:173.705 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:55:51:173.820 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:55:51:173.920 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:55:51:173.965 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:55:51:174.051 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:55:51:252.228 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:55:51:252.493 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Connect : Start Connect timer 10.0.0.42:8086 2025-03-20 Thu 01:55:51:252.608 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:51:252.642 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: SANDESH: Send FAILED: 1742435751252327 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:51:252.653 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:51:252.683 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:51:252.845 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: SANDESH: Send FAILED: 1742435751252593 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 3 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200853 calls = 24 average_bytes = 8368 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 12 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 3 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 83, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 26, _iter59->first = sent _iter59->second = 29, ] ] ] 2025-03-20 Thu 01:55:51:252.871 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:51:255.189 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:45132::10.0.0.42:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:55:51:255.232 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:45132::10.0.0.42:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:55:51:255.258 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:45132::10.0.0.42:8086(-1):EvTcpConnected 2025-03-20 Thu 01:55:51:255.341 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:55:51:255.377 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:55:51:255.428 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: ClientInit 2025-03-20 Thu 01:55:51:255.450 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-2.:contrail-query-engine:0:Database count 4 2025-03-20 Thu 01:55:51:255.562 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:51:255.573 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:55:51:255.596 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: SANDESH: Send FAILED: 1742435751255418 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:51:255.616 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:55:51:255.625 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:51:255.631 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:55:51:255.643 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:51:255.649 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:51:255.869 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: SANDESH: Send FAILED: 1742435751255548 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 200853 calls = 24 average_bytes = 8368 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 13 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 3 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 28, _iter59->first = sent _iter59->second = 29, ] ] ] 2025-03-20 Thu 01:55:51:255.897 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398364862208, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:52:502.887 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-03-20 Thu 01:55:52:502.978 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:52:502.990 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:52:503.071 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: SANDESH: Send FAILED: 1742435752502765 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 60 dequeues = 60 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 75 dequeues = 75 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = ClientInit state_since = 1742435751255426 last_state = ClientInit last_event = scm::EvSandeshSend last_event_at = 1742435751255641 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] max_sm_queue_count = 6 ] 2025-03-20 Thu 01:55:52:503.082 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:52:503.091 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:52:503.098 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:52:503.237 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: SANDESH: Send FAILED: 1742435752502860 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 202426 calls = 25 average_bytes = 8097 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 30, _iter59->first = sent _iter59->second = 30, ] ] ] 2025-03-20 Thu 01:55:52:503.295 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:53:255.591 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398330619648, Pid 1]: 10.0.0.42:8086 EvConnectTimerExpired in state ClientInit 2025-03-20 Thu 01:55:53:255.680 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Processing scm::EvConnectTimerExpired in state ClientInit 2025-03-20 Thu 01:55:53:255.720 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: ClientInit : EvConnectTimerExpired 2025-03-20 Thu 01:55:53:256.011 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Return to idle with 5000 2025-03-20 Thu 01:55:53:256.030 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Idle 2025-03-20 Thu 01:55:53:256.129 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:53:256.309 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: SANDESH: Send FAILED: 1742435753255852 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 4 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 202426 calls = 25 average_bytes = 8097 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 30, ] ] ] 2025-03-20 Thu 01:55:53:256.347 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:53:256.360 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:53:256.382 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: SANDESH: Send FAILED: 1742435753255898 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Down description = ClientInit : EvConnectTimerExpired ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:53:256.389 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:53:256.398 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:55:53:256.427 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:53:256.555 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: SANDESH: Send FAILED: 1742435753256114 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 4 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 202426 calls = 25 average_bytes = 8097 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 14 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 32, _iter59->first = sent _iter59->second = 30, ] ] ] 2025-03-20 Thu 01:55:53:256.572 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:56:002.254 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:55:56:002.418 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:55:56:002.469 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:55:56:002.495 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:55:56:002.524 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:55:56:002.831 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:55:56:002.870 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:55:56:002.884 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:55:56:002.896 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:55:56:003.019 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:55:56:172.708 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:55:56:172.867 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:55:56:172.919 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:55:56:172.946 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:55:56:172.956 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:55:56:172.982 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:55:56:173.005 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:55:56:173.049 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:55:56:173.063 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:55:56:173.074 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:55:56:173.722 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:55:56:173.848 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:55:56:173.942 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:55:56:174.001 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:55:56:174.077 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:55:56:174.231 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:55:56:174.269 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:55:56:174.289 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:55:56:174.306 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:55:56:174.399 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:55:58:256.185 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397699380992, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:55:58:256.489 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397699380992, Pid 1]: Connect : Start Connect timer 10.0.0.47:8086 2025-03-20 Thu 01:55:58:256.640 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397699380992, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:58:256.679 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397699380992, Pid 1]: SANDESH: Send FAILED: 1742435758256314 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:58:256.688 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397699380992, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:58:256.698 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397699380992, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:55:58:256.845 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397699380992, Pid 1]: SANDESH: Send FAILED: 1742435758256618 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 4 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 202426 calls = 25 average_bytes = 8097 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 15 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 4 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 84, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 35, _iter59->first = sent _iter59->second = 30, ] ] ] 2025-03-20 Thu 01:55:58:256.885 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397699380992, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:58:259.334 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:35504::10.0.0.47:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:55:58:259.367 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:35504::10.0.0.47:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:55:58:259.375 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:35504::10.0.0.47:8086(-1):EvTcpConnected 2025-03-20 Thu 01:55:58:259.443 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:55:58:259.475 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:55:58:259.525 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: ClientInit 2025-03-20 Thu 01:55:58:259.552 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-2.:contrail-query-engine:0:Database count 5 2025-03-20 Thu 01:55:58:259.646 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:58:259.656 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:55:58:259.694 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SANDESH: Send FAILED: 1742435758259509 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:58:259.702 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:55:58:259.710 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:58:259.717 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:55:58:259.729 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:55:58:259.735 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:55:58:260.276 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SANDESH: Send FAILED: 1742435758259636 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 202426 calls = 25 average_bytes = 8097 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 16 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 5 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 85, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 37, _iter59->first = sent _iter59->second = 30, ] ] ] 2025-03-20 Thu 01:55:58:260.310 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:55:58:786.476 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.12:35504::10.0.0.47:8086(-1) < Read failed due to error system 104 : Connection reset by peer src/contrail-common/io/tcp_session.cc 507 2025-03-20 Thu 01:55:58:786.529 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session 10.0.0.12:35504::10.0.0.47:8086(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:55:58:786.552 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:35504::10.0.0.47:8086(-1) OnSessionEvent TCP Connection Closed 2025-03-20 Thu 01:55:58:786.560 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:35504::10.0.0.47:8086(-1):EvTcpClose 2025-03-20 Thu 01:55:58:787.858 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvTcpClose in state ClientInit 2025-03-20 Thu 01:55:58:787.889 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: ClientInit : EvTcpClose 2025-03-20 Thu 01:55:58:788.195 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Return to idle with 5000 2025-03-20 Thu 01:55:58:788.209 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Idle 2025-03-20 Thu 01:55:58:788.301 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:58:788.467 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: SANDESH: Send FAILED: 1742435758787978 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 39, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:55:58:788.495 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:55:58:788.506 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:58:788.528 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: SANDESH: Send FAILED: 1742435758788163 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Down description = ClientInit : EvTcpClose ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:55:58:788.535 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:55:58:788.544 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:55:58:788.553 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:55:58:788.696 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: SANDESH: Send FAILED: 1742435758788288 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 17 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 39, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:55:58:788.712 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:01:004.019 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:01:004.194 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:01:004.263 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:01:004.308 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:01:004.347 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:01:004.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:01:004.645 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:01:004.665 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:01:004.693 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:01:004.817 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:01:172.985 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:01:173.117 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:01:173.169 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:01:173.195 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:01:173.208 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:01:173.291 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:01:173.335 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:01:173.383 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:01:173.405 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:01:173.422 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:01:174.331 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:01:174.429 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:01:174.458 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:01:174.486 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:01:174.507 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:01:174.823 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:01:174.916 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:01:174.974 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:01:175.024 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:01:175.052 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:03:788.405 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397716174592, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:56:03:788.623 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397716174592, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-03-20 Thu 01:56:03:788.740 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397716174592, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:56:03:788.769 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397716174592, Pid 1]: SANDESH: Send FAILED: 1742435763788487 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:03:788.778 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397716174592, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:03:788.787 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397716174592, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:56:03:788.927 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397716174592, Pid 1]: SANDESH: Send FAILED: 1742435763788722 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 18 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 42, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:03:790.311 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397716174592, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:03:788.966 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 01:56:03:790.364 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 01:56:03:790.427 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 01:56:03:790.454 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 01:56:03:790.604 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:56:03:790.644 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Idle 2025-03-20 Thu 01:56:03:790.710 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:03:790.858 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: SANDESH: Send FAILED: 1742435763790546 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 7 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 87, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 44, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:03:790.881 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:03:790.891 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:03:790.913 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: SANDESH: Send FAILED: 1742435763790579 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:03:790.920 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:03:790.928 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:56:03:790.954 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:03:791.232 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: SANDESH: Send FAILED: 1742435763790699 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 19 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 44, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:03:791.296 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:06:004.697 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:06:004.850 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:06:004.911 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:06:004.961 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:06:004.994 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:06:005.269 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:06:005.302 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:06:005.318 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:06:005.330 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:06:005.342 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:06:173.233 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:06:173.394 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:06:173.462 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:06:173.488 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:06:173.513 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:06:173.555 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:06:173.591 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:06:173.623 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:06:173.637 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:06:173.648 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:06:174.882 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:06:174.992 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:06:175.041 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:06:175.073 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:06:175.105 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:06:175.416 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:06:175.483 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:06:175.496 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:06:175.508 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:06:175.548 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:11:005.453 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:11:005.646 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:11:005.693 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:11:005.736 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:11:005.777 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:11:005.993 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:11:006.040 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:11:006.063 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:11:006.081 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:11:006.115 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:11:173.558 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:11:173.726 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:11:173.778 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:11:173.805 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:11:173.816 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:11:173.841 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:11:173.866 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:11:173.895 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:11:173.908 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:11:173.919 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:11:175.477 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:11:175.601 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:11:175.630 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:11:175.653 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:11:175.674 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:11:176.180 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:11:176.357 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:11:176.384 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:11:176.398 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:11:176.411 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:11:790.817 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:56:11:791.317 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Connect : Start Connect timer 10.0.0.42:8086 2025-03-20 Thu 01:56:11:791.430 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:56:11:791.473 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435771791106 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:11:791.495 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:11:791.504 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:56:11:791.723 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435771791413 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 20 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 27 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 47, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:11:791.736 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 01:56:11:791.795 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 01:56:11:791.765 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:11:791.838 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 01:56:11:791.855 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 01:56:11:791.967 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:56:11:792.000 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Idle 2025-03-20 Thu 01:56:11:792.058 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:11:792.259 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435771791918 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 8 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 88, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 49, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:11:792.283 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:11:792.298 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:11:792.321 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435771791943 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:11:792.331 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:11:792.339 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:56:11:792.359 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:11:792.519 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: SANDESH: Send FAILED: 1742435771792049 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 21 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 28 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 49, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:11:792.539 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398373259008, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:16:006.135 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:16:006.354 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:16:006.391 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:16:006.439 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:16:006.463 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:16:006.816 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:16:006.854 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:16:006.916 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:16:006.931 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:16:006.998 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:16:173.914 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:16:174.182 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:16:174.306 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:16:174.347 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:16:174.362 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:16:174.411 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:16:174.446 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:16:174.490 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:16:174.508 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:16:174.524 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:16:176.262 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:16:176.397 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:16:176.441 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:16:176.472 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:16:176.508 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:16:176.965 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:16:176.997 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:16:177.016 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:16:177.053 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:16:177.199 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:21:006.919 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:21:007.100 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:21:007.142 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:21:007.166 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:21:007.187 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:21:007.524 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:21:007.564 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:21:007.648 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:21:007.901 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:21:007.934 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:21:174.747 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:21:174.930 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:21:175.032 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:21:175.061 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:21:175.071 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:21:175.111 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:21:175.149 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:21:175.197 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:21:175.218 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:21:175.235 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:21:177.034 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:21:177.716 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:21:177.803 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:21:177.863 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:21:177.918 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:21:178.098 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:21:178.134 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:21:178.239 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:21:178.394 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:21:183.808 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:22:503.368 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:22:503.508 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: SANDESH: Send FAILED: 1742435782503186 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 83 dequeues = 83 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 110 dequeues = 110 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742435771791999 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742435771792358 ] max_sm_queue_count = 6 ] 2025-03-20 Thu 01:56:22:503.525 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:22:503.536 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:22:503.706 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: SANDESH: Send FAILED: 1742435782503317 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 52, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:22:503.747 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:26:007.622 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:26:007.791 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:26:007.830 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:26:007.937 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:26:007.985 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:26:008.216 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:26:008.298 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:26:008.397 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:26:009.044 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:26:009.069 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:26:175.101 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:26:175.430 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:26:175.536 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:26:175.586 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:26:175.602 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:26:175.668 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:26:175.701 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:26:175.731 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:26:175.855 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:26:175.875 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:26:178.159 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:26:178.259 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:26:178.408 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:26:178.463 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:26:178.809 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:26:178.834 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:26:179.042 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:26:179.099 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:26:183.947 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:26:184.511 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:27:482.337 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:27:482.466 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: SANDESH: Send FAILED: 1742435787482275 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-03-20 Thu 01:56:27:482.482 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397707777792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-03-20 Thu 01:56:27:792.196 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:56:27:792.480 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Connect : Start Connect timer 10.0.0.47:8086 2025-03-20 Thu 01:56:27:792.682 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:56:27:792.717 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: SANDESH: Send FAILED: 1742435787792297 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:27:792.744 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:27:792.753 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:56:27:792.933 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: SANDESH: Send FAILED: 1742435787792666 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 22 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 55, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:27:792.950 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:27:793.227 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 01:56:27:793.297 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 01:56:27:793.375 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 01:56:27:793.388 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 01:56:27:793.532 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:56:27:793.577 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Idle 2025-03-20 Thu 01:56:27:793.637 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:27:793.800 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: SANDESH: Send FAILED: 1742435787793471 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 9 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 89, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:27:793.821 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:27:793.835 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:27:793.866 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: SANDESH: Send FAILED: 1742435787793503 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:27:793.880 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:27:793.891 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:56:27:793.902 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:27:794.030 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: SANDESH: Send FAILED: 1742435787793627 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 23 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 57, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:27:794.043 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398360663808, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:31:008.346 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:31:008.543 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:31:008.607 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:31:009.001 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:31:009.047 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:31:009.078 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:31:009.099 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:31:009.151 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:31:009.475 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:31:009.510 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:31:175.650 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:31:175.881 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:31:175.961 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:31:176.000 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:31:176.013 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:31:176.055 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:31:176.093 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:31:176.137 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:31:176.155 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:31:176.171 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:31:178.903 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:31:179.095 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:31:179.141 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:31:179.175 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:31:179.548 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:31:179.585 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:31:179.600 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:31:179.613 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:31:184.598 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:31:185.216 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:36:009.117 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:36:009.332 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:36:009.376 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:36:009.509 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:36:009.561 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:36:009.814 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:36:009.856 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:36:009.878 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:36:009.900 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:36:009.917 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:36:176.056 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:36:176.307 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:36:176.363 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:36:176.395 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:36:176.405 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:36:176.432 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:36:176.456 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:36:176.487 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:36:176.500 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:36:176.510 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:36:179.629 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:36:179.738 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:36:179.768 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:36:179.813 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:36:180.286 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:36:180.343 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:36:180.357 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:36:180.370 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:36:185.323 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:36:185.853 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:41:009.899 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:41:010.035 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:41:010.072 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:41:010.096 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:41:010.256 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:41:010.518 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:41:010.545 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:41:010.627 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:41:010.641 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:41:010.653 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:41:176.439 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:41:176.597 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:41:176.667 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:41:176.699 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:41:176.709 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:41:176.740 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:41:176.764 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:41:176.793 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:41:176.806 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:41:176.830 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:41:180.368 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:41:180.476 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:41:180.525 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:41:180.565 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:41:181.071 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:41:181.106 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:41:181.121 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:41:181.134 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:41:185.991 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:41:186.599 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:46:010.669 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:46:010.831 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:46:010.898 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:46:010.946 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:46:011.011 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:46:011.256 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:46:011.301 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:46:011.323 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:46:011.445 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:46:011.470 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:46:176.763 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:46:176.930 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:46:176.984 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:46:177.010 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:46:177.020 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:46:177.061 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:46:177.086 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:46:177.134 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:46:177.150 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:46:177.161 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:46:181.157 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:46:181.309 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:46:181.346 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:46:181.369 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:46:181.754 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:46:181.785 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:46:182.162 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:46:186.678 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:46:187.205 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:46:191.674 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:51:011.354 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:51:011.524 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:51:011.592 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:51:011.634 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:51:011.889 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:51:011.953 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:51:011.981 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:51:011.993 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:51:012.127 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:51:012.313 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:51:177.113 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:51:177.300 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:51:177.472 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:51:177.504 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:51:177.515 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:51:177.548 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:51:177.572 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:51:177.602 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:51:177.615 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:51:177.626 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:51:181.810 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:51:181.904 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:51:182.189 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:51:182.422 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:51:182.581 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:51:182.744 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:51:187.272 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:51:188.041 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:51:191.750 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:51:192.213 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:52:503.645 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-03-20 Thu 01:56:52:503.754 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398369060608, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:52:503.864 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398369060608, Pid 1]: SANDESH: Send FAILED: 1742435812503508 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 91 dequeues = 91 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 3 dequeues = 3 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 8 dequeues = 8 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 121 dequeues = 121 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742435787793576 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742435787793900 ] max_sm_queue_count = 6 ] 2025-03-20 Thu 01:56:52:503.890 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398369060608, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:52:503.901 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398369060608, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:52:504.063 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398369060608, Pid 1]: SANDESH: Send FAILED: 1742435812503627 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 35 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 60, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:52:504.082 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398369060608, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:56:012.220 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:56:56:012.410 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:56:56:012.461 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:56:56:012.513 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:56:56:012.561 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:56:56:013.565 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:56:56:013.617 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:56:56:013.640 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:56:56:013.659 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:56:56:013.678 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:56:56:177.552 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:56:56:177.710 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:56:56:177.778 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:56:56:177.809 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:56:56:177.819 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:56:56:177.846 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:56:56:177.869 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:56:56:177.898 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:56:56:177.911 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:56:56:177.921 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:56:56:182.497 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:56:56:182.743 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:56:56:182.902 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:56:56:183.039 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:56:56:183.082 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:56:56:183.419 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:56:56:188.097 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:56:56:188.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:56:56:192.289 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:56:56:192.943 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:56:57:793.750 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:56:57:794.047 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-03-20 Thu 01:56:57:794.066 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 01:56:57:794.102 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 01:56:57:794.157 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:56:57:794.187 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: SANDESH: Send FAILED: 1742435817793853 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:57:794.196 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:57:794.206 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 01:56:57:794.213 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 01:56:57:794.326 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:56:57:794.358 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Idle 2025-03-20 Thu 01:56:57:794.426 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:57:794.625 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: SANDESH: Send FAILED: 1742435817794144 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 24 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 62, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:57:794.661 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:57:794.676 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:57:794.873 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: SANDESH: Send FAILED: 1742435817794276 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 10 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 90, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:57:794.894 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:56:57:794.904 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:57:794.929 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: SANDESH: Send FAILED: 1742435817794304 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 01:56:57:794.939 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:56:57:794.948 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:56:57:795.019 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:56:57:795.230 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: SANDESH: Send FAILED: 1742435817794413 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 25 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 37 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 63, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:56:57:795.273 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:01:015.951 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:57:01:016.111 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:57:01:016.158 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:57:01:016.192 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:57:01:016.222 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:57:01:016.794 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:57:01:016.882 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:57:01:016.903 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:57:01:016.920 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:57:01:016.949 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:57:01:177.850 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:57:01:178.024 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:57:01:178.098 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:57:01:178.133 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:57:01:178.148 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:57:01:178.191 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:57:01:178.226 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:57:01:178.280 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:57:01:178.299 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:57:01:178.323 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:57:01:183.098 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:57:01:183.205 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:57:01:183.458 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:57:01:184.897 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:57:01:184.936 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:57:01:184.952 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:57:01:188.619 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:57:01:189.156 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:57:01:192.995 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:57:01:193.511 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:57:06:016.916 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:57:06:017.054 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:57:06:017.112 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:57:06:017.148 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:57:06:017.177 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:57:06:017.500 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:57:06:017.549 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:57:06:017.566 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:57:06:017.584 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:57:06:017.600 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:57:06:178.173 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:57:06:178.357 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:57:06:178.429 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:57:06:178.457 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:57:06:178.466 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:57:06:178.507 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:57:06:178.531 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:57:06:178.560 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:57:06:178.574 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:57:06:178.589 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:57:06:185.902 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:57:06:186.015 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:57:06:186.048 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:57:06:186.382 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:57:06:186.441 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:57:06:186.457 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:57:06:189.216 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:57:06:189.603 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:57:06:193.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:57:06:194.012 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:57:11:017.604 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:57:11:017.754 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:57:11:017.797 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:57:11:017.825 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:57:11:017.863 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:57:11:018.167 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:57:11:018.205 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:57:11:018.223 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:57:11:018.281 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:57:11:018.318 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:57:11:178.525 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:57:11:178.703 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:57:11:178.755 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:57:11:178.778 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:57:11:178.788 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:57:11:178.814 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:57:11:178.838 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:57:11:178.872 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:57:11:178.885 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:57:11:178.895 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:57:11:186.491 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:57:11:186.640 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:57:11:186.698 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:57:11:187.059 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:57:11:187.110 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:57:11:187.129 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:57:11:189.768 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:57:11:190.430 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:57:11:194.082 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:57:11:194.648 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:57:16:018.303 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:57:16:018.473 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:57:16:018.509 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:57:16:018.767 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:57:16:018.817 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:57:16:018.898 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:57:16:018.951 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:57:16:019.273 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:57:16:019.312 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:57:16:019.332 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:57:16:178.839 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:57:16:179.002 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:57:16:179.066 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:57:16:179.089 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:57:16:179.099 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:57:16:179.126 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:57:16:179.151 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:57:16:179.181 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:57:16:179.194 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:57:16:179.205 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:57:16:187.163 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:57:16:187.354 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:57:16:187.401 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:57:16:187.672 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:57:16:187.718 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:57:16:187.784 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:57:16:190.528 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:57:16:191.038 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:57:16:194.737 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:57:16:195.289 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:57:21:019.016 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:57:21:019.167 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:57:21:019.330 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:57:21:019.395 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:57:21:019.438 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:57:21:019.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22d61d0 2025-03-20 Thu 01:57:21:019.611 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dc5f0 2025-03-20 Thu 01:57:21:019.675 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd340 2025-03-20 Thu 01:57:21:019.716 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dd870 2025-03-20 Thu 01:57:21:019.794 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dcca0 2025-03-20 Thu 01:57:21:179.132 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:57:21:179.282 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:57:21:179.331 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e06a0 2025-03-20 Thu 01:57:21:179.356 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22dfd60 2025-03-20 Thu 01:57:21:179.366 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:57:21:179.392 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:57:21:179.415 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:57:21:179.444 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22df420 2025-03-20 Thu 01:57:21:179.457 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22deae0 2025-03-20 Thu 01:57:21:179.467 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22de1a0 2025-03-20 Thu 01:57:21:187.741 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:57:21:187.847 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:57:21:187.874 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:57:21:188.515 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e0fe0 2025-03-20 Thu 01:57:21:188.551 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e1920 2025-03-20 Thu 01:57:21:188.565 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2ba0 2025-03-20 Thu 01:57:21:191.084 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:57:21:191.542 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e2260 2025-03-20 Thu 01:57:21:195.366 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:57:21:195.928 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: -1 0x22e34e0 2025-03-20 Thu 01:57:22:504.129 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:22:504.366 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: SANDESH: Send FAILED: 1742435842503875 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 6 dequeues = 6 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 98 dequeues = 98 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 9 dequeues = 9 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 131 dequeues = 131 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742435817794357 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742435817795015 ] max_sm_queue_count = 6 ] 2025-03-20 Thu 01:57:22:504.389 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:22:504.404 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:22:504.632 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: SANDESH: Send FAILED: 1742435842504018 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 26 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 1 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 79 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 40 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 91, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 67, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:57:22:504.667 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:26:019.651 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22d61d0 2025-03-20 Thu 01:57:26:019.838 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dc5f0 2025-03-20 Thu 01:57:26:019.900 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd340 2025-03-20 Thu 01:57:26:019.944 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dd870 2025-03-20 Thu 01:57:26:019.987 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dcca0 2025-03-20 Thu 01:57:26:020.331 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22d61d0 2025-03-20 Thu 01:57:26:020.345 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:020.375 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:020.411 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22dc5f0 2025-03-20 Thu 01:57:26:020.418 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:020.431 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:020.442 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22dd870 2025-03-20 Thu 01:57:26:020.449 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:020.460 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:020.471 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22dd340 2025-03-20 Thu 01:57:26:020.478 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:020.490 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:020.501 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22dcca0 2025-03-20 Thu 01:57:26:020.520 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:020.531 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:020.902 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:020.929 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:020.959 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:021.084 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:021.094 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:021.106 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:021.114 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:021.405 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:26:021.446 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SANDESH: Send FAILED: 1742435846021146 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.12:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:26:021.454 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:26:021.475 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:021.506 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:021.529 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:021.552 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:021.572 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:021.586 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:021.612 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:021.636 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:179.408 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e06a0 2025-03-20 Thu 01:57:26:179.643 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22dfd60 2025-03-20 Thu 01:57:26:179.711 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e06a0 2025-03-20 Thu 01:57:26:179.721 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:179.746 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:179.757 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22dfd60 2025-03-20 Thu 01:57:26:179.762 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:179.770 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:179.776 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22df420 2025-03-20 Thu 01:57:26:179.817 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22deae0 2025-03-20 Thu 01:57:26:179.855 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22de1a0 2025-03-20 Thu 01:57:26:179.934 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22df420 2025-03-20 Thu 01:57:26:179.941 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:179.951 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:179.958 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22deae0 2025-03-20 Thu 01:57:26:179.964 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:179.975 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:179.998 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22de1a0 2025-03-20 Thu 01:57:26:180.003 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:180.010 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:180.041 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:180.051 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:180.063 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:180.072 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:180.131 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:180.248 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:180.264 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:180.273 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:180.290 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:180.294 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:26:180.361 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: SANDESH: Send FAILED: 1742435846180161 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:26:180.372 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397690984192, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:26:180.302 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:180.711 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:180.740 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:180.755 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:180.784 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:180.807 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:188.617 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e0fe0 2025-03-20 Thu 01:57:26:188.745 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e1920 2025-03-20 Thu 01:57:26:188.815 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2ba0 2025-03-20 Thu 01:57:26:189.206 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e0fe0 2025-03-20 Thu 01:57:26:189.230 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:189.264 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:189.300 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e1920 2025-03-20 Thu 01:57:26:189.306 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:189.315 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:189.446 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e2ba0 2025-03-20 Thu 01:57:26:189.453 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:189.463 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:189.916 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:189.966 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:189.996 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:190.026 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:190.038 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:190.051 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:190.060 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:190.135 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:190.152 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:190.161 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:26:190.219 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: SANDESH: Send FAILED: 1742435846190092 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:26:190.233 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:26:191.607 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e2260 2025-03-20 Thu 01:57:26:192.159 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e2260 2025-03-20 Thu 01:57:26:192.176 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:192.192 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:192.711 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:192.733 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:192.768 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:26:195.978 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RedisAsyncConnection::RAC_Reconnect initiated 0x22e34e0 2025-03-20 Thu 01:57:26:196.487 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: RAC_Connect status: 0 0x22e34e0 2025-03-20 Thu 01:57:26:196.505 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: Connected to REDIS... 2025-03-20 Thu 01:57:26:196.523 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 01:57:26:196.925 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 01:57:26:196.951 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 01:57:26:196.980 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 01:57:27:483.320 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:27:483.423 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: SANDESH: Send FAILED: 1742435847483237 [SYS_NOTICE]: QEDbStatsUve: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] ] 2025-03-20 Thu 01:57:27:483.435 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397711976192, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: QEDbStatsUve 2025-03-20 Thu 01:57:27:794.517 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:57:27:794.793 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Connect : Start Connect timer 10.0.0.42:8086 2025-03-20 Thu 01:57:27:794.985 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:57:27:795.035 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: SANDESH: Send FAILED: 1742435847794601 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.42:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:27:795.048 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:27:795.073 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:57:27:795.225 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: -(-1) OnSessionEvent TCP Connect Failed 2025-03-20 Thu 01:57:27:795.268 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: -(-1):EvTcpConnectFail 2025-03-20 Thu 01:57:27:795.379 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: SANDESH: Send FAILED: 1742435847794941 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.42:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 29 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 42 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 73, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:57:27:795.410 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:27:795.423 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvTcpConnectFail in state Connect 2025-03-20 Thu 01:57:27:795.431 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Connect : EvTcpConnectFail 2025-03-20 Thu 01:57:27:795.551 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: TCP [SYS_ERR]: TcpSessionMessageLog: Session -(-1) > Shutdown failed due to error: Transport endpoint is not connected src/contrail-common/io/tcp_session.cc 312 2025-03-20 Thu 01:57:27:795.588 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Idle 2025-03-20 Thu 01:57:27:795.673 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:27:795.927 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: SANDESH: Send FAILED: 1742435847795507 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 11 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 151, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 75, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:57:27:795.956 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:27:795.971 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:27:796.005 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: SANDESH: Send FAILED: 1742435847795530 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Connect : EvTcpConnectFail ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:27:796.015 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:27:796.028 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 01:57:27:796.057 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:27:796.286 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: SANDESH: Send FAILED: 1742435847795659 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 30 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 43 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 75, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:57:27:796.313 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:52:504.367 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-03-20 Thu 01:57:52:504.432 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:52:504.500 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SANDESH: Send FAILED: 1742435872504157 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 109 dequeues = 109 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 145 dequeues = 145 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742435847795587 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742435847796055 ] max_sm_queue_count = 6 ] 2025-03-20 Thu 01:57:52:504.525 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:52:504.534 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 01:57:52:504.673 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SANDESH: Send FAILED: 1742435872504345 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 45 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 78, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:57:52:504.686 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:57:795.934 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Processing scm::EvIdleHoldTimerExpired in state Idle 2025-03-20 Thu 01:57:57:796.285 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Connect : Start Connect timer 10.0.0.47:8086 2025-03-20 Thu 01:57:57:796.405 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:57:57:796.443 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SANDESH: Send FAILED: 1742435877796010 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Idle : EvIdleHoldTimerExpired -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:57:796.452 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 01:57:57:796.461 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 01:57:57:796.618 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: SANDESH: Send FAILED: 1742435877796382 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 5 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 31 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 47 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 12 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 152, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 80, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:57:57:796.632 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397686785792, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 01:57:57:799.215 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:56138::10.0.0.47:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 280 2025-03-20 Thu 01:57:57:799.252 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:56138::10.0.0.47:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 01:57:57:799.263 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398525327488, Pid 1]: 10.0.0.12:56138::10.0.0.47:8086(-1):EvTcpConnected 2025-03-20 Thu 01:57:57:806.728 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 01:57:57:806.761 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 01:57:57:806.817 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: ClientInit 2025-03-20 Thu 01:57:57:806.835 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-2.:contrail-query-engine:0:Database count 6 2025-03-20 Thu 01:57:57:806.952 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:57:57:806.981 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 01:57:57:807.011 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: SANDESH: Send FAILED: 1742435877806806 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 01:57:57:807.022 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 01:57:57:807.037 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:57:57:807.046 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 01:57:57:807.062 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 01:57:57:807.071 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 01:57:57:807.231 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: SANDESH: Send FAILED: 1742435877806940 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 82, _iter59->first = sent _iter59->second = 31, ] ] ] 2025-03-20 Thu 01:57:57:807.269 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397695182592, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 01:57:57:873.577 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: OnMessage control in state: ClientInit 2025-03-20 Thu 01:57:57:873.639 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-03-20 Thu 01:57:57:873.649 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-03-20 Thu 01:57:57:873.673 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Received Ctrl Message with size 0 2025-03-20 Thu 01:57:57:873.906 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 6 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 2033 calls = 4 average_bytes = 508 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 203999 calls = 26 average_bytes = 7846 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] sm_queue_count = 0 sm_stats= [ ev_stats= [ [ [ event = scm::EvCollectorUpdate enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvConnectTimerExpired enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvIdleHoldTimerExpired enqueues = 7 dequeues = 7 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshMessageRecv enqueues = 2 dequeues = 2 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvSandeshSend enqueues = 109 dequeues = 109 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvStart enqueues = 1 dequeues = 1 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpClose enqueues = 4 dequeues = 4 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnectFail enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpConnected enqueues = 5 dequeues = 5 enqueue_fails = 0 dequeue_fails = 0 ], [ event = scm::EvTcpDeleteSession enqueues = 10 dequeues = 10 enqueue_fails = 0 dequeue_fails = 0 ], [ event = enqueues = 145 dequeues = 145 enqueue_fails = 0 dequeue_fails = 0 ], ] ] state = Idle state_since = 1742435847795587 last_state = Idle last_event = scm::EvSandeshSend last_event_at = 1742435847796055 ] session_stats= [ num_recv_msg = 0 num_recv_msg_fail = 0 num_recv_fail = 0 num_send_msg = 1 num_send_msg_fail = 0 num_send_buffer_fail = 0 num_wait_msgq_enqueue = 0 num_wait_msgq_dequeue = 0 num_write_ready_cb_error = 0 ] session_rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] session_tx_socket_stats= [ bytes = 1573 calls = 1 average_bytes = 1573 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 32 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 4 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 5 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 12 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 48 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 82, _iter59->first = sent _iter59->second = 31, ] ] max_sm_queue_count = 6 proxy seq 60 2025-03-20 Thu 01:57:57:873.940 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:57:57:873.964 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.47:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 41 2025-03-20 Thu 01:57:57:873.976 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:57:57:874.009 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2. table_info= [ [ ] ] errors= [ write_tablespace_fails = 0 read_tablespace_fails = 0 write_table_fails = 0 read_table_fails = 0 write_column_fails = 0 write_batch_column_fails = 0 read_column_fails = 0 ] stats_info= [ [ ] ] cql_stats= [ requests_one_minute_rate = 0.131848 stats= [ total_connections = 6 available_connections = 6 exceeded_pending_requests_water_mark = 0 exceeded_write_bytes_water_mark = 0 ] errors= [ connection_timeouts = 0 pending_request_timeouts = 0 request_timeouts = 0 ] ] proxy seq 5 2025-03-20 Thu 01:57:57:874.018 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 01:57:57:874.025 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 01:57:57:874.057 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Established 2025-03-20 Thu 01:57:58:300.914 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: OnMessage control in state: Established 2025-03-20 Thu 01:57:58:300.972 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshMessageRecv in state Established 2025-03-20 Thu 01:57:58:300.981 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Established : EvSandeshMessageRecv 2025-03-20 Thu 01:57:58:301.023 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Received Ctrl Message with size 3 2025-03-20 Thu 01:57:58:301.035 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 with seqno 61, total = 0 2025-03-20 Thu 01:57:58:301.046 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 with seqno 43, total = 0 2025-03-20 Thu 01:57:58:301.055 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 with seqno 5, total = 0 2025-03-20 Thu 01:57:58:301.062 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 01:58:52:505.501 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398326421248, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-03-20 Thu 01:59:52:507.852 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-03-20 Thu 02:00:52:508.888 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398326421248, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-03-20 Thu 02:01:52:510.025 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140397703579392, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-03-20 Thu 02:02:52:510.645 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398381655808, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-03-20 Thu 02:03:06:017.894 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvStop in state Established 2025-03-20 Thu 02:03:06:017.993 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Idle 2025-03-20 Thu 02:03:06:018.105 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvTcpDeleteSession in state Idle 2025-03-20 Thu 02:03:06:018.132 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Processing scm::EvSandeshSend in state Idle 2025-03-20 Thu 02:03:06:018.344 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: SANDESH: Send FAILED: 1742436186018083 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 6 pid = 1 http_port = 8091 start_time = 1742435572284855 collector_name = collector_ip = 10.0.0.47:8086 collector_list= [ [ (*_iter40) = 10.0.0.42:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.12:8086, ] ] rx_socket_stats= [ bytes = 4480 calls = 8 average_bytes = 560 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 501766 calls = 55 average_bytes = 9123 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 13 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 33 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QEDbStatsUve [ messages_sent = 10 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 2 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 139 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlClientToServer [ messages_sent = 6 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshCtrlServerToClient [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 34 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 49 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 13 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 153, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 84, _iter59->first = sent _iter59->second = 63, ] ] ] 2025-03-20 Thu 02:03:06:018.366 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140398377457408, Pid 1]: Wrong state: Idle for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 02:03:36:180.885 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: http-server-port 8091 2025-03-20 Thu 02:03:36:181.097 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Max-tasks 16 2025-03-20 Thu 02:03:36:181.106 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Max-slice 100 2025-03-20 Thu 02:03:36:181.111 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Collectors 10.0.0.42:8086 2025-03-20 Thu 02:03:36:181.116 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Collectors 10.0.0.12:8086 2025-03-20 Thu 02:03:36:181.120 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Collectors 10.0.0.47:8086 2025-03-20 Thu 02:03:36:181.146 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: ROLE : Generator 2025-03-20 Thu 02:03:36:181.152 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: MODULE : contrail-query-engine 2025-03-20 Thu 02:03:36:181.157 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: SOURCE : cn-jenkins-deploy-platform-ansible-os-2923-2. 2025-03-20 Thu 02:03:36:181.168 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: NODE TYPE : Database 2025-03-20 Thu 02:03:36:181.173 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: INSTANCE ID : 0 2025-03-20 Thu 02:03:36:181.177 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: HTTP SERVER PORT : 8091 2025-03-20 Thu 02:03:36:181.183 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: System Log Send Rate Limit: 0 -> 0 2025-03-20 Thu 02:03:36:181.961 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: HTTP Introspect Init 2025-03-20 Thu 02:03:36:182.117 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: TCP [SYS_DEBUG]: TcpServerMessageLog: Server 0.0.0.0:8091 Initialization complete src/contrail-common/io/tcp_server.cc 137 2025-03-20 Thu 02:03:36:182.176 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Sandesh Http Server Port: 8091 2025-03-20 Thu 02:03:36:182.214 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: NOT Writing http_port 8091TO : /tmp/contrail-query-engine.0.http_port 2025-03-20 Thu 02:03:36:182.223 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: CONNECT TO COLLECTOR: 1 2025-03-20 Thu 02:03:36:182.322 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Idle 2025-03-20 Thu 02:03:36:182.483 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: No Client: 1742436216182408 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Idle successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742436216182332 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 0, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 02:03:36:182.595 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: Logging: DISABLED -> ENABLED 2025-03-20 Thu 02:03:36:182.615 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: SANDESH: Logging: LEVEL: [ INVALID ] -> [ SYS_DEBUG ] log4level: [ NOTSET ] -> [ DEBUG ] 2025-03-20 Thu 02:03:36:182.646 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Cassandra Servers: 10.0.0.42:9042 10.0.0.12:9042 10.0.0.47:9042 2025-03-20 Thu 02:03:36:182.762 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Processing scm::EvStart in state Idle 2025-03-20 Thu 02:03:36:182.847 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Disconnect 2025-03-20 Thu 02:03:36:182.918 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Processing scm::EvCollectorUpdate in state Disconnect 2025-03-20 Thu 02:03:36:183.157 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Connect : Start Connect timer 10.0.0.12:8086 2025-03-20 Thu 02:03:36:183.240 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:03:36:183.261 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: SANDESH: Send FAILED: 1742436216182838 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 0.0.0.0:0, ] ] status = Down description = Idle : EvStart -> Disconnect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-03-20 Thu 02:03:36:183.269 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 02:03:36:183.277 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:03:36:183.358 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: SANDESH: Send FAILED: 1742436216182908 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742436216182332 collector_name = collector_ip = 0.0.0.0:0 collector_list= [ [ ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 02:03:36:183.375 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 02:03:36:183.384 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:03:36:183.450 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: SANDESH: Send FAILED: 1742436216182979 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Disconnect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742436216182332 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.42:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 02:03:36:183.461 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 02:03:36:183.469 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:03:36:183.483 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: SANDESH: Send FAILED: 1742436216182998 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], ] ] flag_infos= [ [ ] ] description = Number of connections:1, Expected:3 Missing: Database,Redis-Query:Query ], ] ] ] 2025-03-20 Thu 02:03:36:183.491 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 02:03:36:183.502 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:03:36:183.606 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: SANDESH: Send FAILED: 1742436216183220 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = Connect successful_connections = 0 pid = 1 http_port = 8091 start_time = 1742436216182332 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.42:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 1, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 0, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 02:03:36:183.626 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: SandeshModuleClientTrace 2025-03-20 Thu 02:03:36:244.890 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: Older table does not exist. will query only the new table src/contrail-analytics/contrail-query-engine/query.cc 1037 2025-03-20 Thu 02:03:36:251.583 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271124563712, Pid 1]: Processing scm::EvSandeshSend in state Connect 2025-03-20 Thu 02:03:36:251.645 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271124563712, Pid 1]: SANDESH: Send FAILED: 1742436216248820 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Number of connections:2, Expected:3 Missing: Redis-Query:Query ], ] ] ] 2025-03-20 Thu 02:03:36:251.654 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271124563712, Pid 1]: Wrong state: Connect for event: EvSandeshSend message: NodeStatusUVE 2025-03-20 Thu 02:03:36:252.377 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:40942::10.0.0.12:8086(-1) < Active session connection complete src/contrail-common/io/tcp_session.cc 282 2025-03-20 Thu 02:03:36:252.400 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: 10.0.0.12:40942::10.0.0.12:8086(-1) OnSessionEvent TCP Connected 2025-03-20 Thu 02:03:36:252.407 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: 10.0.0.12:40942::10.0.0.12:8086(-1):EvTcpConnected 2025-03-20 Thu 02:03:36:252.435 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc95c70 2025-03-20 Thu 02:03:36:252.441 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.455 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.459 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Processing scm::EvTcpConnected in state Connect 2025-03-20 Thu 02:03:36:252.472 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc964d0 2025-03-20 Thu 02:03:36:252.509 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.517 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.526 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc96d50 2025-03-20 Thu 02:03:36:252.530 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.550 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.559 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc975d0 2025-03-20 Thu 02:03:36:252.563 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.570 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.578 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc97e50 2025-03-20 Thu 02:03:36:252.583 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.590 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.12 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.598 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc58240 2025-03-20 Thu 02:03:36:252.603 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.610 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.619 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc58a00 2025-03-20 Thu 02:03:36:252.623 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.630 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.646 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc59210 2025-03-20 Thu 02:03:36:252.651 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.663 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.671 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc5a310 2025-03-20 Thu 02:03:36:252.675 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.683 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.690 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc986d0 2025-03-20 Thu 02:03:36:252.694 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.701 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 0 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.709 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc59a90 2025-03-20 Thu 02:03:36:252.713 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.723 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.42 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.731 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc98f50 2025-03-20 Thu 02:03:36:252.736 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.743 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 1 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.750 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc997d0 2025-03-20 Thu 02:03:36:252.755 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.762 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 2 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.770 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc9a050 2025-03-20 Thu 02:03:36:252.774 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.781 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 3 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:252.935 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: RAC_Connect status: 0 0xc9a8d0 2025-03-20 Thu 02:03:36:252.944 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: Connected to REDIS... 2025-03-20 Thu 02:03:36:252.952 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp.. UP 4 With Redis:10.0.0.47 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 973 2025-03-20 Thu 02:03:36:253.375 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:253.403 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:253.467 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:253.489 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:253.502 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:253.588 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:253.603 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:253.855 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:253.875 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:253.893 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:253.906 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.12:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:253.958 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:253.985 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:254.009 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:254.033 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:254.102 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:254.116 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:254.154 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:254.167 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:254.184 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:254.206 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:254.223 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:254.248 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:254.284 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:254.306 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:254.323 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:254.333 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.42:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:254.347 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:254.358 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:254.371 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:254.381 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:254.433 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:254.466 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:254.484 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:254.501 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:256.219 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:258.291 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:258.323 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:258.336 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB2 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:258.354 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:258.363 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:258.399 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: UP ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1025 2025-03-20 Thu 02:03:36:258.415 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: In ConnectCallbackProcess..10.0.0.47:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 1032 2025-03-20 Thu 02:03:36:258.437 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB3 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:258.456 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: ConnUp SetCB4 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 682 2025-03-20 Thu 02:03:36:252.502 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Connect : EvTcpConnected : Cancelling Connect timer 2025-03-20 Thu 02:03:36:258.534 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: ClientInit 2025-03-20 Thu 02:03:36:258.558 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Sending Ctrl Message for cn-jenkins-deploy-platform-ansible-os-2923-2.:contrail-query-engine:0:Database count 1 2025-03-20 Thu 02:03:36:258.656 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:03:36:258.666 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 02:03:36:258.689 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: SANDESH: Send FAILED: 1742436216253454 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Down description = ], ] ] flag_infos= [ [ ] ] description = Collector, Redis-Query:Query connection down ], ] ] ] 2025-03-20 Thu 02:03:36:258.696 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 02:03:36:258.704 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:03:36:258.710 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 02:03:36:258.731 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: SANDESH: Send FAILED: 1742436216253943 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.42:6379::Down.10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 02:03:36:258.739 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 02:03:36:258.756 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:03:36:258.764 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 02:03:36:258.788 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: SANDESH: Send FAILED: 1742436216254457 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = 10.0.0.47:6379::Down. ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 02:03:36:258.797 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 02:03:36:258.807 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:03:36:258.816 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 02:03:36:258.831 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: SANDESH: Send FAILED: 1742436216254517 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Disconnect : EvCollectorUpdate -> Connect ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 02:03:36:258.836 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 02:03:36:258.844 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:03:36:258.849 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: ClientInit : EvSandeshSend : NodeStatusUVE 2025-03-20 Thu 02:03:36:258.863 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: SANDESH: Send FAILED: 1742436216258524 [SYS_NOTICE]: NodeStatusUVE: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] ] 2025-03-20 Thu 02:03:36:258.873 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Received UVE message in wrong state : NodeStatusUVE 2025-03-20 Thu 02:03:36:258.880 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:03:36:258.886 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: ClientInit : EvSandeshSend : SandeshCtrlClientToServer 2025-03-20 Thu 02:03:36:258.910 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Processing scm::EvSandeshSend in state ClientInit 2025-03-20 Thu 02:03:36:258.916 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: ClientInit : EvSandeshSend : SandeshModuleClientTrace 2025-03-20 Thu 02:03:36:259.124 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: SANDESH: Send FAILED: 1742436216258643 [SYS_NOTICE]: SandeshModuleClientTrace: data= [ name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742436216182332 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.42:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] ] 2025-03-20 Thu 02:03:36:259.141 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: Received UVE message in wrong state : SandeshModuleClientTrace 2025-03-20 Thu 02:03:36:262.885 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: OnMessage control in state: ClientInit 2025-03-20 Thu 02:03:36:262.943 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: Processing scm::EvSandeshMessageRecv in state ClientInit 2025-03-20 Thu 02:03:36:262.977 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: ClientInit : EvSandeshMessageRecv 2025-03-20 Thu 02:03:36:263.003 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: Received Ctrl Message with size 0 2025-03-20 Thu 02:03:36:263.189 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: SyncUVE Syncing ObjectGeneratorInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2.:Database:contrail-query-engine:0 client_info= [ status = ClientInit successful_connections = 1 pid = 1 http_port = 8091 start_time = 1742436216182332 collector_name = collector_ip = 10.0.0.12:8086 collector_list= [ [ (*_iter40) = 10.0.0.12:8086, (*_iter40) = 10.0.0.47:8086, (*_iter40) = 10.0.0.42:8086, ] ] rx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] tx_socket_stats= [ bytes = 0 calls = 0 average_bytes = 0 blocked_duration = 00:00:00 blocked_count = 0 average_blocked_duration = errors = 0 ] ] msg_type_agg= [ [ _iter58->first = NodeStatusUVE [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = QELog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 61 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = SandeshModuleClientTrace [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 1 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 3 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 0 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpServerMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], _iter58->first = TcpSessionMessageLog [ messages_sent = 0 messages_sent_dropped_no_queue = 0 messages_sent_dropped_no_client = 0 messages_sent_dropped_no_session = 0 messages_sent_dropped_queue_level = 0 messages_sent_dropped_client_send_failed = 0 messages_sent_dropped_session_not_connected = 0 messages_sent_dropped_header_write_failed = 0 messages_sent_dropped_write_failed = 0 messages_sent_dropped_wrong_client_sm_state = 0 messages_sent_dropped_validation_failed = 0 messages_sent_dropped_rate_limited = 0 messages_sent_dropped_sending_disabled = 1 messages_sent_dropped_sending_to_syslog = 0 ], ] ] tx_msg_agg= [ [ _iter59->first = dropped_client_send_failed _iter59->second = 0, _iter59->first = dropped_header_write_failed _iter59->second = 0, _iter59->first = dropped_no_client _iter59->second = 1, _iter59->first = dropped_no_queue _iter59->second = 0, _iter59->first = dropped_no_session _iter59->second = 0, _iter59->first = dropped_queue_level _iter59->second = 0, _iter59->first = dropped_rate_limited _iter59->second = 0, _iter59->first = dropped_sending_disabled _iter59->second = 63, _iter59->first = dropped_sending_to_syslog _iter59->second = 0, _iter59->first = dropped_session_not_connected _iter59->second = 0, _iter59->first = dropped_validation_failed _iter59->second = 0, _iter59->first = dropped_write_failed _iter59->second = 0, _iter59->first = dropped_wrong_client_sm_state _iter59->second = 6, _iter59->first = sent _iter59->second = 0, ] ] proxy seq 5 2025-03-20 Thu 02:03:36:263.226 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: SyncAllMaps for ModuleClientState:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 02:03:36:263.283 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: SyncUVE Syncing ObjectDatabaseInfo val name = cn-jenkins-deploy-platform-ansible-os-2923-2. process_status= [ [ [ module_id = contrail-query-engine instance_id = 0 state = Non-Functional connection_infos= [ [ [ type = Collector name = server_addrs= [ [ (*_iter6) = 10.0.0.12:8086, ] ] status = Initializing description = Connect : EvTcpConnected ], [ type = Database name = server_addrs= [ [ (*_iter6) = 10.0.0.42:9042, (*_iter6) = 10.0.0.12:9042, (*_iter6) = 10.0.0.47:9042, ] ] status = Up description = ], [ type = Redis-Query name = Query server_addrs= [ [ (*_iter6) = 10.0.0.42:6379, (*_iter6) = 10.0.0.12:6379, (*_iter6) = 10.0.0.47:6379, ] ] status = Up description = ], ] ] flag_infos= [ [ ] ] description = Collector connection down ], ] ] proxy seq 8 2025-03-20 Thu 02:03:36:263.308 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: SyncAllMaps for NodeStatus:0 period 0 cycle 0 without seqno , total = 1 2025-03-20 Thu 02:03:36:263.318 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: SyncAllMaps for QEDbStats:0 period 0 cycle 0 without seqno , total = 0 2025-03-20 Thu 02:03:36:263.328 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:0 period 60 cycle 0 without seqno , total = 0 2025-03-20 Thu 02:03:36:263.366 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: Established 2025-03-20 Thu 02:04:36:183.170 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 1 without seqno , total = 0 2025-03-20 Thu 02:05:36:183.951 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270746203904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 2 without seqno , total = 0 2025-03-20 Thu 02:06:36:185.120 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 3 without seqno , total = 0 2025-03-20 Thu 02:07:36:185.883 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271116166912, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 4 without seqno , total = 0 2025-03-20 Thu 02:08:36:186.584 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 5 without seqno , total = 0 2025-03-20 Thu 02:09:36:187.406 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271086778112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 6 without seqno , total = 0 2025-03-20 Thu 02:10:36:188.069 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271120365312, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 7 without seqno , total = 0 2025-03-20 Thu 02:11:36:188.648 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271095174912, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 8 without seqno , total = 0 2025-03-20 Thu 02:12:36:189.328 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271095174912, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 9 without seqno , total = 0 2025-03-20 Thu 02:13:36:189.965 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270742005504, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 10 without seqno , total = 0 2025-03-20 Thu 02:14:36:190.621 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270729410304, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 11 without seqno , total = 0 2025-03-20 Thu 02:15:36:191.372 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270733608704, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 12 without seqno , total = 0 2025-03-20 Thu 02:16:36:192.185 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 13 without seqno , total = 0 2025-03-20 Thu 02:17:12:126.041 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:8091::10.0.0.12:52844 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-03-20 Thu 02:17:36:192.888 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 14 without seqno , total = 0 2025-03-20 Thu 02:17:36:731.734 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:8091::10.0.0.12:48920 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-03-20 Thu 02:17:52:676.810 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:8091::10.0.0.12:34992 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-03-20 Thu 02:18:07:937.909 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:8091::10.0.0.12:38872 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-03-20 Thu 02:18:36:193.723 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 15 without seqno , total = 0 2025-03-20 Thu 02:18:41:328.538 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:8091::10.0.0.12:58800 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-03-20 Thu 02:19:36:194.307 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 16 without seqno , total = 0 2025-03-20 Thu 02:20:01:647.813 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:8091::10.0.0.42:59716 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-03-20 Thu 02:20:24:461.438 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.42 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-03-20 Thu 02:20:24:462.482 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID e193d900-0531-11f0-b009-00000a00002a src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-03-20 Thu 02:20:24:462.545 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QEQueryLog: e193d900-0531-11f0-b009-00000a00002a :0:json query is: { end_time: 1742437224402585 , enqueue_time: 1742437224460530 , query_metadata: {"enqueue_time": 1742437224460431} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "87d6e16b-1ec1-45c8-b000-792372cbfae7", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:24:462.845 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-03-20 Thu 02:20:24:462.873 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-03-20 Thu 02:20:24:463.114 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271086778112, Pid 1]: [SYS_INFO]: QEQueryLog: e193d900-0531-11f0-b009-00000a00002a :0:json query is: { end_time: 1742437224402585 , enqueue_time: 1742437224460530 , query_metadata: {"enqueue_time": 1742437224460431} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "87d6e16b-1ec1-45c8-b000-792372cbfae7", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:24:463.971 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for e193d900-0531-11f0-b009-00000a00002a , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-03-20 Thu 02:20:24:510.717 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:510.739 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:510.846 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:510.853 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:510.868 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:510.910 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:510.920 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:510.928 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:510.936 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:510.944 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:510.951 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:510.997 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 3 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:511.006 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 3 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:510.788 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:510.890 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:511.049 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:511.080 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:511.084 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:511.104 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QEQueryLog: e193d900-0531-11f0-b009-00000a00002a :0:json query is: { end_time: 1742437224402585 , enqueue_time: 1742437224460530 , query_metadata: {"enqueue_time": 1742437224460431} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectVMTable" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "87d6e16b-1ec1-45c8-b000-792372cbfae7", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:24:511.104 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:511.115 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:511.162 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:511.176 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:511.192 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:511.209 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:511.225 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:511.236 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:511.265 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 2 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:511.280 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:24:511.295 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 2 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:511.308 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: e193d900-0531-11f0-b009-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:24:511.408 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QEQueryLog: e193d900-0531-11f0-b009-00000a00002a ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id 87d6e16b-1ec1-45c8-b000-792372cbfae7 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:24:511.680 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QEQueryLog: e193d900-0531-11f0-b009-00000a00002a ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id 87d6e16b-1ec1-45c8-b000-792372cbfae7 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:24:511.734 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QEQueryLog: e193d900-0531-11f0-b009-00000a00002a ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 87d6e16b-1ec1-45c8-b000-792372cbfae7 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:24:512.103 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QEQueryLog: e193d900-0531-11f0-b009-00000a00002a ObjectVMTable :1:Column ObjectTypeName1 Table ObjectVMTable object_id 87d6e16b-1ec1-45c8-b000-792372cbfae7 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:24:512.118 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QEQueryLog: e193d900-0531-11f0-b009-00000a00002a ObjectVMTable :0:Column ObjectTypeName1 Table ObjectVMTable object_id 87d6e16b-1ec1-45c8-b000-792372cbfae7 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:24:512.510 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-03-20 Thu 02:20:24:512.633 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 5 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-03-20 Thu 02:20:24:512.735 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":5} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-03-20 Thu 02:20:24:514.071 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: Finished: QID e193d900-0531-11f0-b009-00000a00002a Table ObjectVMTable Time(ms) 51 RedisTime(ms) 0 MergeTime(ms) 0 Rows 5 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-03-20 Thu 02:20:24:514.159 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-03-20 Thu 02:20:24:514.217 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271111968512, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2923-2. table = ObjectVMTable query_stats= [ qid = e193d900-0531-11f0-b009-00000a00002a where = [[{"name": "ObjectId", "value": "87d6e16b-1ec1-45c8-b000-792372cbfae7", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 46,0, 47,0, 47,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 51 rows = 5 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-03-20 Thu 02:20:27:560.720 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: StartPipeline on : Redis:10.0.0.42 Port:6379 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 773 2025-03-20 Thu 02:20:27:561.794 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: Got Query to prepare for QID e36ca870-0531-11f0-bbf0-00000a00002a src/contrail-analytics/contrail-query-engine/query.cc 1166 2025-03-20 Thu 02:20:27:561.835 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QEQueryLog: e36ca870-0531-11f0-bbf0-00000a00002a :0:json query is: { end_time: 1742437227544571 , enqueue_time: 1742437227559852 , query_metadata: {"enqueue_time": 1742437227559736} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500:ctest-vn22-97468500", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:27:562.041 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: Chunks: 3 Need Merge: 0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 880 2025-03-20 Thu 02:20:27:562.059 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: Getting Least Loaded Conn as :0 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 931 2025-03-20 Thu 02:20:27:562.123 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: [SYS_INFO]: QELog: Starting Pipeline for e36ca870-0531-11f0-bbf0-00000a00002a , 1 conn, 16 tasks src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 940 2025-03-20 Thu 02:20:27:562.210 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270742005504, Pid 1]: [SYS_INFO]: QEQueryLog: e36ca870-0531-11f0-bbf0-00000a00002a :0:json query is: { end_time: 1742437227544571 , enqueue_time: 1742437227559852 , query_metadata: {"enqueue_time": 1742437227559736} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500:ctest-vn22-97468500", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:27:595.555 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:595.601 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:595.610 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:595.626 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:595.633 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:595.640 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:595.646 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:595.653 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:595.660 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:595.666 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:599.397 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:599.443 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:599.452 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:599.459 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:599.466 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:599.473 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:599.479 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:599.493 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:599.500 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:599.509 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 1 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:599.771 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QEQueryLog: e36ca870-0531-11f0-bbf0-00000a00002a ObjectRoutingInstance :1:Column ObjectTypeName1 Table ObjectRoutingInstance object_id default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500:ctest-vn22-97468500 src/contrail-analytics/contrail-query-engine/select.cc 1023 2025-03-20 Thu 02:20:27:601.053 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:601.083 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:601.092 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:601.113 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:601.120 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:601.127 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:601.134 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:601.140 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:601.147 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a UNION between tables of sizes 0 and 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 60 2025-03-20 Thu 02:20:27:601.154 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: e36ca870-0531-11f0-bbf0-00000a00002a Resulting size of set 0 src/contrail-analytics/contrail-query-engine/set_operation.cc 67 2025-03-20 Thu 02:20:27:601.237 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: [SYS_INFO]: QEQueryLog: e36ca870-0531-11f0-bbf0-00000a00002a :0:json query is: { end_time: 1742437227544571 , enqueue_time: 1742437227559852 , query_metadata: {"enqueue_time": 1742437227559736} , select_fields: ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] , start_time: 1742437205862627 , table: "ObjectRoutingInstance" , table_schema: [{"name": "MessageTS", "datatype": "int", "index": false, "suffixes": null}, {"name": "ObjectId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Source", "datatype": "string", "index": true, "suffixes": null}, {"name": "ModuleId", "datatype": "string", "index": true, "suffixes": null}, {"name": "Messagetype", "datatype": "string", "index": true, "suffixes": null}, {"name": "ObjectLog", "datatype": "string", "index": false, "suffixes": null}, {"name": "SystemLog", "datatype": "string", "index": false, "suffixes": null}] , where: [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500:ctest-vn22-97468500", "op": 1, "value2": null}]] } src/contrail-analytics/contrail-query-engine/query.cc 467 2025-03-20 Thu 02:20:27:601.513 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: Will Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 520 2025-03-20 Thu 02:20:27:601.590 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: Did Jsonify #rows 1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 528 2025-03-20 Thu 02:20:27:601.638 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QELog: QE Query Result is {"progress":100, "lines":1, "count":1} src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 563 2025-03-20 Thu 02:20:27:602.086 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: Finished: QID e36ca870-0531-11f0-bbf0-00000a00002a Table ObjectRoutingInstance Time(ms) 40 RedisTime(ms) 0 MergeTime(ms) 0 Rows 1 EnQ-delay1 src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 644 2025-03-20 Thu 02:20:27:602.164 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270737807104, Pid 1]: [SYS_INFO]: QELog: Result 1 , 1 conn src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc 676 2025-03-20 Thu 02:20:27:602.113 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: [SYS_INFO]: QueryPerfInfo: name = cn-jenkins-deploy-platform-ansible-os-2923-2. table = ObjectRoutingInstance query_stats= [ qid = e36ca870-0531-11f0-bbf0-00000a00002a where = [[{"name": "ObjectId", "value": "default-domain:ctest-AnalyticsBasicTestSanity-41612179:ctest-vn22-97468500:ctest-vn22-97468500", "op": 1, "value2": null}]] select = { "select" : ["ObjectId", "Source", "ObjectLog", "SystemLog", "ModuleId", "Messagetype", "MessageTS"] } post = time_span = 8 error = None chunks = 3 chunk_where_time = 33,0, 36,0, 38,0, chunk_select_time = 0,0, 0,0, 0,0, chunk_postproc_time = 0,0, 0,0, 0,0, chunk_merge_time = final_merge_time = 0 time = 40 rows = 1 enq_delay = 1 ] file = src/contrail-analytics/contrail-query-engine/QEOpServerProxy.cc line = 636 2025-03-20 Thu 02:20:36:196.343 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 17 without seqno , total = 0 2025-03-20 Thu 02:21:36:197.340 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 18 without seqno , total = 0 2025-03-20 Thu 02:22:36:198.282 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 19 without seqno , total = 0 2025-03-20 Thu 02:23:36:199.314 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 20 without seqno , total = 0 2025-03-20 Thu 02:24:36:200.174 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271107770112, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 21 without seqno , total = 0 2025-03-20 Thu 02:25:36:200.799 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140270725211904, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 22 without seqno , total = 0 2025-03-20 Thu 02:26:36:201.935 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 23 without seqno , total = 0 2025-03-20 Thu 02:27:14:633.340 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:8091::10.0.0.42:51074 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256 2025-03-20 Thu 02:27:36:202.740 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271090976512, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 24 without seqno , total = 0 2025-03-20 Thu 02:28:36:203.951 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271103571712, Pid 1]: SyncAllMaps for UserDefinedLogStatistic:2 period 60 cycle 25 without seqno , total = 0 2025-03-20 Thu 02:28:39:502.945 UTC cn-jenkins-deploy-platform-ansible-os-2923-2 [Thread 140271272425600, Pid 1]: TCP [SYS_DEBUG]: TcpSessionMessageLog: Session 10.0.0.12:8091::10.0.0.12:43776 > Passive session Accept complete src/contrail-common/io/tcp_session.cc 256